How broad is that claim? Mapping Generalisation in NLP Research

Scientific communication often relies on generalisations to extrapolate findings, but these claims are frequently semantically ambiguous and can lead to over-interpretation. The authors address the need for a systematic way to identify and categorise these claims by introducing NLPGenX, a taxonomy that labels scientific outcome sentences based on their level of generality and framing. To operationalise this, they develop NLPGenA, an LLM-driven framework that uses a series of linguistic classifiers to automatically annotate sentences. The authors validate this framework using human expert annotations and apply it to a corpus of 4.5K NLP papers, resulting in NLPGenS, a large-scale dataset of 180K annotated sentences.

The authors find that generalisations are prevalent in NLP literature, with the majority of outcome-reporting sentences falling into the 'framed/non-generic' category. Their analysis reveals that generalised claims are significantly more likely to contain hedging and vague descriptors compared to non-generalised claims. Furthermore, they observe a statistically significant correlation between the use of certain framed generalisations and higher citation counts in specific venues like ACL 2017, suggesting that effective scoping of claims may be associated with higher-impact research. The study also highlights that the distribution of generalisation types varies across different NLP sub-domains and has shown a slight upward trend over recent years.

This work provides a critical tool for researchers to audit the language used in scientific reporting. By automatically identifying over-generalised claims, the community can better detect potential misrepresentations of scientific findings. The taxonomy and dataset offer a foundation for future research into the pragmatics of scientific writing, helping to distinguish between well-supported, scoped claims and potentially misleading generalisations that lack sufficient evidence.