Are Near-Tied LLM Rankings Robust to Family-DIF-Guided Benchmark Recomposition?

Public LLM leaderboards often rank models based on tiny score differences, sometimes less than one percentage point. This paper investigates whether these near-tied rankings are robust to the specific selection of items included in a benchmark. The authors ask: if we adjust for systematic differences in how model families perform on specific items (differential item functioning, or DIF), do the rankings of closely matched models remain stable?

To address this, the authors adapt psychometric methods to LLM evaluation. They use a family-blind spectral approximation to multidimensional item-response theory (MIRT) to identify items that exhibit low residual differential item functioning (DIF) across model families. By using owner-disjoint cross-fitting, they construct "low-DIF" anchor subtests that are balanced for source and easiness. They then compare these results against matched-random subtests—which control for subtest length and composition—to isolate whether the observed ranking reversals are driven by targeted composition sensitivity or simply by the inherent noise of using smaller subtests.

Across five major benchmarks, the authors find that while global rankings remain highly correlated, the specific ordering of near-tied models is fragile. In four of the five benchmarks, 30.9–47.1% of cross-family pairs initially within one percentage point of each other reversed their relative order under low-DIF scoring. This reversal rate significantly exceeded the matched-random control, suggesting that the near-tie rankings are sensitive to benchmark composition. The fifth benchmark, WinoGrande, did not show this reliable excess. The authors also performed a blinded content audit to see if these reversals were tied to specific semantic features (like reasoning type or domain knowledge), but found no consistent, replicable content-based explanation.