Accuracy and Order Sensitivity Diverge Under Label-Free Strategies

Multiple-choice question (MCQ) benchmarks are a dominant format for evaluating large language models, but they conflate true knowledge with sensitivity to option order and formatting. This paper investigates whether preventing models from seeing option labels while committing to an answer removes positional bias and improves evaluation accuracy. The authors evaluate two main label-free mitigation strategies across six proprietary and open-weight models on the MMLU and ARC-Challenge benchmarks. The first strategy is two-stage prompting, where the model generates a free-text answer without options before matching it to a label. The second strategy is independent hypothesis scoring, which prompts the model separately for each option in isolation to construct a positionally unbiased selection.

The experiments show that neither label-free strategy reliably improves end-to-end accuracy. Two-stage prompting reduces accuracy in almost all tested model-benchmark pairs, driven both by parse failures during the matching step and by the information bottleneck of withholding options in the first stage. A 2x2 diagnostic grid reveals that performance drops sharply when options are hidden, whereas showing options in the first stage recovers much of the lost performance. Independent hypothesis scoring likewise fails to consistently raise accuracy despite eliminating positional influence by construction. In contrast, simple cyclic permutation improves accuracy across most model-benchmark pairs without requiring complex prompting modifications or logit access.

To measure positional effects, the authors track recall standard deviation (RStd) and a direct per-question flip rate under cyclic permutation. The results demonstrate that reducing positional sensitivity does not automatically translate into accuracy gains. For instance, some models exhibit substantially lower flip rates under debiasing strategies while suffering concurrent drops in task accuracy. These findings indicate that MCQ brittleness stems from deeper limitations in how models evaluate and compare choices rather than merely surface-level option-order artifacts.