Hyunjong Ok, Minjae Park, Jaeho Lee
9 min
When evaluating language models on multiple-choice questions, researchers often find that simply reordering the options changes the model's answer. This phenomenon is typically viewed through two lenses: selection bias (a preference for specific option labels like A, B, or C) and order sensitivity (inconsistency across orderings). This paper reconciles these perspectives by providing an exact mathematical identity: the pooled selection-rate standard deviation is the product of the flip rate and the positional skew.
The authors define two metrics that do not require gold labels:
By testing 41 model checkpoints across 12 benchmarks, the authors demonstrate that these two factors capture distinct behaviors. The flip rate is strongly linked to model accuracy and output uncertainty (models that are less sure of their answers flip more often). In contrast, the positional skew effectively tracks the model's inherent preference for specific option tokens (the "option-symbol prior").
The paper evaluates common mitigation strategies and reveals that they affect these two factors differently:
The authors find that while both strategies improve consistency, they may leave different residual problems. Aggregation, for instance, can lower the flip rate by settling questions on the same wrong option. Consequently, the authors argue that reporting both factors alongside accuracy is essential for a complete audit of model stability and correctness.
Reordering multiple-choice options can change a model's answer, an effect studied as selection bias (a preference for certain option letters) and as order sensitivity (changes in answers across option orderings). We connect these perspectives through the pooled selection-rate standard deviation (pooled SD), which measures how unevenly option letters are selected across questions and orderings. When every option of each question appears equally often at each position, the pooled SD is exactly the product of two label-free factors: the flip rate and the positional skew. The flip rate is the fraction of questions whose chosen option changes across orderings; the positional skew is the standard deviation of option-letter selection rates among those questions. Across 41 checkpoints and 12 benchmarks, the flip rate tracks accuracy and output uncertainty, while the positional skew tracks the estimated option-symbol prior. Prior correction for selection bias mainly reduces the positional skew. Aggregating each ordering with its reversal, a mitigation for order sensitivity, mainly reduces the flip rate. Both reduce the range of accuracy across correct-answer letters by similar amounts, but aggregation leaves fewer model-rank reversals. Applying prior correction before aggregation reduces both factors and further narrows this accuracy range. However, aggregation lowers the flip rate partly by settling questions on the same wrong option. We therefore recommend reporting both factors to distinguish mitigation effects, alongside accuracy to distinguish stability from correctness.
Sam: So the letter stays fixed, but the answer does not. When heat moves away from A, the model chooses something else.
Alex: That counts as a flip. The authors’ flip rate is the share of questions whose chosen content changes across the evaluated orderings. Keeping the letter A is not consistency if A keeps holding different content.
Sam: What if the model follows heat wherever it moves? How does that contribute to the pooled score?
Alex: Then its answers land evenly across the letters. And the same is true if it consistently follows a wrong option. Under balanced orderings, any stable content choice contributes an even spread.
Sam: Which means only the changing answers can make the overall letter counts uneven. Is that where the mathematical result comes from?
Alex: Yes. The pooled score equals the flip rate multiplied by positional skew. Positional skew means how unevenly the flipping questions’ answers land on the letters. This is an arithmetic identity under balanced orderings, not a proposed explanation of the model’s internals.
Sam: So identical scores could describe a model that changes often with a mild letter preference, or rarely with a strong preference.
Alex: The paper shows both possibilities. It also makes a useful warning explicit: changing answers can favour different letters and cancel out overall. An even pooled distribution does not establish that individual answers are stable.
Sam: How broadly did they test whether these factors capture different behaviour?
Alex: They evaluated 41 open-weight model checkpoints, spanning several families and both pretrained and post-trained models. Their evaluation covered 12 benchmarks, with different numbers of answer options. The main answer readout selected the letter with the highest next-token probability.
Sam: And what separated the factors empirically, rather than just mathematically?
Alex: Across model–benchmark pairs, the flip rate was strongly associated with lower accuracy. Within a pair, flips concentrated on questions where the leading answer probabilities were close. Positional skew instead tracked an estimated background preference for particular answer letters.
Sam: Accurate models have less room to change answers without becoming wrong. Could that alone explain the accuracy relationship?
Alex: The authors checked that restriction. The association persisted where the accuracy-based limit imposed no extra constraint, and when accuracy was measured on separate items. But they do not claim that internal uncertainty causes flipping; they measured associations in the outputs.
Sam: Now the practical comparison. What do the existing fixes actually do?
Alex: Prior correction divides letter probabilities by an estimated background letter preference. The study tests PriDe and a batch-calibration variant. The other fix combines each ordering with its reversal, choosing the content with the largest product of its probabilities across those queries.
Sam: One adjusts each query’s letter preferences. The other combines evidence about the same content from different positions. Did they change the same factor?
Alex: Mostly not. Prior correction reduced positional skew, with little net change in the flip rate. Combining an ordering with its reversal mainly reduced flips, and reduced some skew too.
Sam: If both improve a bias score, what difference would an evaluator actually notice?
Alex: Consider the placement range: how far accuracy moves when all correct answers are placed under one letter versus another. In the shared comparison, the baseline range was about seventeen percentage points. Reversal aggregation brought it to about eight.
Sam: Did prior correction provide comparable protection, or was aggregation simply better across the board?
Alex: The corrections narrowed that range by similar amounts, but reversal aggregation left fewer model-rank reversals across orderings. Applying PriDe before aggregation narrowed the placement range further, to about five percentage points. The combined approach reduced both factors, but did not clearly improve ranking stability beyond aggregation alone.
Sam: We started with consistently wrong answers. How much of the apparent stability improvement came from that?
Alex: About two in five questions whose flips were resolved by reversal aggregation settled on a wrong option. Average accuracy still improved, but the share of consistently wrong questions rose by more. Usually, the settled wrong option was already the most frequently chosen content across the queried orderings.
Sam: So it is not simply inventing new wrong choices. It often locks in a distractor the model already favoured.
Alex: That describes the observed transition, not its cause. There’s also a scope limit: stability under the tested rotations does not guarantee stability under every permutation. And these experiments mainly use deterministic answers on mostly English benchmarks.
Sam: Would the same measurement work for a model that gives different answers when you repeat an identical prompt?
Alex: Not cleanly. In the authors’ closed-model checks, repeated identical prompts produced substantial disagreement. A measured flip rate there mixes reordering effects with ordinary response variation. The authors therefore do not compare those factors directly with the deterministic open-weight results.
Sam: Who should read the full document, and where should they begin?
Alex: Anyone designing multiple-choice evaluations or comparing bias mitigations. Start with “Two factors in pooled bias statistics” for the balanced-ordering argument. Then read “Residual harms and composition” for the comparison between corrections, aggregation, ranking changes and stable wrong answers.
Sam: And what should someone outside model evaluation carry back to their own work?
Alex: Keep correctness beside stability, and ask what an improved summary score actually changed. The line to remember is this: fewer changing answers can mean more answers that stay wrong.
Sam: That’s a useful distinction to keep.