AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing

As LLMs are increasingly used as judges to evaluate other models, their inherent biases—such as position, verbosity, and self-preference—often lead to evaluations that diverge from human judgment. This paper addresses the challenge of auditing these LLM-as-a-judge decisions to determine when they should be trusted, corrected, or sent for human review, particularly when human annotation resources are limited.

The authors propose AURA (Adaptive Uncertainty-aware Refinement), a framework that treats the agreement between an LLM judge and human preference as a latent variable. Instead of relying on a fixed set of clean labels, AURA iteratively updates its estimates of judge reliability. The process involves three main components: training a human-consistency scorer, propagating reliable evidence through a sparse transport-based mechanism, and selectively querying human labels for the most uncertain or influential comparisons. By treating the "trusted" and "uncertain" groups as dynamic quantities, the method avoids the pitfalls of standard positive-unlabeled (PU) learning, which often assumes a static set of clean labels.

AURA provides a stable, iterative loop that converges to a self-consistent state where judge-human agreement is better estimated. Theoretical analysis demonstrates that the algorithm exhibits linear convergence and numerical stability. Empirical evaluations on both synthetic and real-world pairwise LLM-answer data show that AURA effectively recovers latent human-consistency structures, significantly improving the accuracy of judge auditing compared to baseline methods while maintaining high annotation efficiency.