A Judge Should Know What Changed: Construct Validity for LLM-as-a-Judge Evaluation

This paper investigates whether LLM-as-a-judge evaluators actually measure the constructs they are intended to assess, rather than merely correlating with surface-level features. The authors formalize construct validity as a two-dimensional profile: invariance (S), the ability to ignore construct-preserving edits, and construct sensitivity (R), the ability to detect construct-changing edits. They test 7 LLM judges across 4 domains using a set of minimal interventions. Crucially, they decompose construct-changing edits into two axes: scope (the breadth of situations a claim covers) and strength (the level of commitment to those situations). Directionality for these edits was established by human annotators, and generation, verification, and judging were performed by disjoint model families to prevent circularity.

• Reliability vs. Validity: Judges are highly invariant (average S = 0.945) but significantly less sensitive (average R = 0.319) to construct changes. This confirms that an instrument can be reliable—consistent in its verdicts—without being valid.
• The Scope/Strength Asymmetry: Judges are consistently more sensitive to scope edits than to strength edits, with a performance gap of +0.121. This asymmetry holds across all 7 tested models, regardless of parameter scale or reasoning capability.
• Validation Power: The authors audit five public label sets and find that a surface-only predictor can reproduce 55%–67% of the labels. This suggests that many existing benchmarks are solvable without actually evaluating the construct, explaining why previous studies may have missed the sensitivity failures reported here.

This work demonstrates that high agreement scores in LLM evaluation are not evidence of validity. By showing that judges are "blind" to strength-based overreach, the authors provide a mechanism for the paradox where prompting models for accuracy can inadvertently increase overgeneralization. The study provides a framework for researchers to report both invariance and sensitivity, rather than relying on scalar agreement metrics that mask structural failures in evaluation.