How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

The Reliability Crisis in AI Safety Evaluation

Most research on LLM jailbreaks relies on automated judges to determine whether a model has successfully produced harmful content. These judges typically fall into two categories: dedicated safety classifiers (like Llama-Guard) or general-purpose LLMs prompted to act as graders. This paper demonstrates that these judges are rarely validated against human ground truth, leading to inflated or deflated attack-success rates (ASR) that do not reflect actual model behavior.

Divergent Failure Modes

The study compares these two judge families using 596 human-labeled completions. The results reveal that they fail in fundamentally different ways. Dedicated classifiers tend to over-flag, exhibiting high recall but lower precision, which systematically inflates ASR. Conversely, LLM-as-judge models demonstrate high precision but erratic, often very low, recall. Because these judges disagree significantly with human annotators and with each other, the reported ASR for the same set of model responses can vary wildly depending on which judge is used.

Vulnerability to Manipulation

Beyond baseline calibration issues, the judges are highly susceptible to adversarial manipulation. The author tests "content-preserving wrappers"—benign framing (such as a polite refusal prefix or educational context) added to harmful responses. These wrappers fool every LLM-judge tested, flipping their verdicts between 57% and 100% of the time. While dedicated classifiers are more robust to these surface-level tricks, they remain vulnerable to white-box gradient-based attacks (GCG). When an attacker has access to the classifier's weights, they can flip 70% of confident true positives even with a small optimization budget. A human audit confirmed that in every case where a judge was "fooled" by these attacks, the harmful content remained entirely intact.

Recommendations for Researchers

To improve the integrity of safety evaluations, the paper proposes three mandatory practices: reporting the judge's precision and recall on a human-labeled subset of the data, reporting ASR corrected for the judge's precision, and performing adversarial checks to ensure the judge is not merely responding to surface-level formatting. Relying on automated judges without these validation steps leads to unreliable metrics that can mislead defense designers and policymakers.