LLM Judges Can Be Too Generous When There Is No Reference Answer

Research Question and Approach

As LLMs are increasingly used as automated judges for open-ended text generation, a critical question arises: can these models reliably evaluate responses when no ground-truth reference is available? The authors investigate this by testing whether LLM judges possess the necessary task knowledge and how their verdicts shift when reference information is introduced.

They employ a two-stage experimental pipeline across English, Arabic, and Telugu. First, they perform calibration experiments to see if judges can distinguish between correct and incorrect ground-truth answers. Second, they conduct sensitivity experiments to measure how judge verdicts change across three conditions: No-Reference (NR), Reference-Visible (RV), and Reference-Compared (RC). By tracking 'decision flips'—instances where a judge changes its verdict from correct to incorrect or vice versa—the authors quantify the impact of reference information on model reliability.

Key Findings

The results reveal that LLM judges are often too generous in no-reference settings, frequently labeling incorrect answers as correct. This tendency is particularly pronounced in lower-resource languages like Telugu. When a reference answer is added to the prompt (the RV setting), the judges often reverse their initial positive verdicts, suggesting that the presence of the reference acts as a necessary corrective signal.

Crucially, the largest shift in behavior occurs simply by making the reference visible, rather than by explicitly instructing the model to perform a comparison. This indicates that judges are highly sensitive to the context provided in the prompt. Comparisons with human annotations confirm that these reference-driven changes generally align with human judgment, implying that the 'no-reference' baseline is often unreliable for high-stakes evaluation.

This study provides a clear warning for practitioners who rely on LLM-as-a-Judge pipelines: without calibration, these models may provide inflated performance metrics. The authors propose a blueprint for researchers to calibrate their judge models using a small, reference-aware sample before deploying them in fully automated, reference-free environments. By systematically checking for task knowledge and sensitivity to reference information, developers can better ensure the integrity of their evaluation systems.