Beyond Benchmarks: Exposing the Hidden Crisis in Bangla Hate Speech Detection

Automatic hate speech (HS) detection is a critical component of online safety, yet current systems for low-resource languages like Bangla often struggle to perform outside of controlled environments. While models such as BanglaBERT achieve high accuracy on standardized benchmark datasets, this study demonstrates that this performance is often an artifact of the data rather than a true mastery of the language. When tested on real-world, noisy social media content from platforms like Facebook and Twitter, these models show a sharp decline in effectiveness, particularly when encountering implicit hate speech—content that relies on cultural context, sarcasm, or emojis rather than explicit slurs.

The researchers evaluated six deep learning architectures, including various combinations of FastText, CNN, LSTM, and BanglaBERT, across three types of datasets: established benchmarks, a merged multi-source dataset, and a newly annotated external validation set. The results reveal a stark contrast: BanglaBERT, which achieved an F1-score of 91.4% on benchmark data, saw its performance drop to 75.3% on real-world data, with accuracy falling to 63.4% specifically for implicit hate speech. Other models, such as FastText + CNN, experienced even more dramatic declines, dropping from 78.0% to 51.2%.

A key finding of this study is the pivotal role of emojis in interpreting intent. The researchers found that emoji-aware preprocessing—treating emojis as meaningful semantic units—improved implicit hate speech detection by up to 12%. Conversely, removing emojis led to a significant performance degradation. Furthermore, the study highlights the danger of over-policing; because these models struggle to distinguish between genuine hate and nuanced political satire or sarcasm, they risk suppressing legitimate freedom of expression. This suggests that current moderation systems are not yet sufficiently culturally grounded to be deployed without human oversight.