From a Word-Level Dictionary to Sentence-Level Semantics: Multilingual Grievance Labelling with Contextual Models

Automated threat assessment often relies on word-level lexicons to identify grievances in online text. However, these tools struggle with the nuances of human language, such as irony, negation, quotation, and speaker attribution. This paper investigates whether these lexical approaches are fundamentally flawed due to circular evaluation methods and whether contextual models—which read the full post rather than just the target sentence—can more accurately identify grievance constructs across multiple languages.

The researchers first analyze the validity of existing grievance lexicons, demonstrating that they are often evaluated on datasets enriched with the very terms they are designed to retrieve, which artificially inflates performance metrics. To address this, they construct a new, non-circular benchmark across five languages (English, Dutch, German, Italian, and French). This benchmark includes three distinct strata: unconditional-random (to estimate base rates), lexicon-positive, and lexicon-negative. They then compare traditional term-matching methods against contextual encoders (mDeBERTa and XLM-R) that process the target sentence within the context of the full post, using a shared 22-construct ontology.

The study reveals that traditional lexical scoring is highly prone to circularity; in one evaluation pool, the lexicon effectively separated the data itself, rendering standard performance metrics misleading. By contrast, the contextual models consistently outperformed target-only models. The most significant gains were observed in the 'lexicon-negative' stratum, where average precision increased from 0.14 to 0.20. The researchers find that reading the full post is essential for resolving complex linguistic phenomena like implicit grievances, cross-sentence meaning, and quoted or condemned speech that word-matching tools typically misidentify.