Selection Shapes the Boundary: A Preregistered Replication of Monotonicity and Label Agreement in Unselected NLI Populations

This study investigates whether linguistic properties—specifically monotonicity—predict human label variation (HLV) in Natural Language Inference (NLI) tasks. Previous research suggested that hypotheses containing non-upward monotonicity operators (such as downward-entailing or non-monotone triggers) were associated with lower annotator agreement. However, that finding was based on the ChaosNLI dataset, which specifically selects items with high disagreement (items where the majority label received only three of five votes). This paper tests whether this boundary holds in the broader, unselected SNLI and MultiNLI development sets.

Using a preregistered methodology, the author applied a frozen, rule-based monotonicity operator tagger to the SNLI and MultiNLI development sets. The study measured the association between monotonicity (purely upward vs. non-upward) and a four-level ordinal agreement scale derived from the original five-label annotations. To ensure the validity of the results, the author conducted robustness checks, including simulated tagger misclassification and a manual audit of 200 items to assess the tagger's accuracy against a defined semantic codebook.

The preregistered prediction failed across all seven tested contrasts. Instead of lower agreement, non-upward items showed slightly higher agreement than upward items, with effect sizes consistently falling below the smallest effect size of interest (SESOI) of 0.10. The author concludes that the previously observed negative boundary is a structural artifact of the selection criteria used in the ChaosNLI dataset rather than an inherent property of the language population. The study highlights that when researchers use selected re-annotation resources, they must explicitly state the selection conditional to avoid misinterpreting dataset-specific patterns as general linguistic phenomena.