ResearchPod Summary
This study investigates the reliability of using star ratings as proxies for textual sentiment in natural language processing (NLP). While star ratings are frequently used as convenient 'weak labels' for training sentiment analysis models, the authors question whether these numerical scores accurately reflect the sentiment expressed in the accompanying written reviews. Using a dataset of 16,156 reviews of Sri Lankan tourism attractions (2010–2023), the researchers employed a transformer-based sentiment pipeline (RoBERTa) to derive textual sentiment independently of the assigned star ratings. They then categorized the resulting mismatches into six directional patterns and used statistical and machine learning models to identify the behavioral and contextual drivers of these discrepancies.
The analysis reveals that 18.6% of reviews exhibit sentiment-rating incongruence, meaning the text and the rating convey different polarities. Rather than being random noise, these mismatches follow a structured typology. The most prevalent patterns are 'Conservative Rater' (where users provide lower ratings than their positive text suggests) and 'Obligatory 5-Star' (where users provide top ratings despite neutral or negative text). The study also finds that incongruence is not uniformly distributed; for instance, museum reviews show higher rates of mismatch compared to national parks. Furthermore, reviewer expertise and review length are significant predictors of these divergences, with expert reviewers being more likely to produce incongruent reviews.
For NLP practitioners, this paper serves as a cautionary tale: treating star ratings as ground-truth labels without validation can introduce systematic bias into sentiment analysis models. By demonstrating that ratings and text capture different dimensions of the user experience, the authors argue for more context-aware sentiment analysis pipelines. This research highlights the necessity of validating weak labels before using them to train or evaluate machine learning systems, particularly in domains where cultural or behavioral factors influence how users assign ratings.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.