A Theory of Post-hoc Debate Judgement

Recent advances in agentic AI leverage debates—where multiple models interact internally or externally to generate pros and cons for different outcomes—to enhance performance and explainability. However, determining the final output from these debates typically relies on post-hoc external judges, such as prompting a large language model. While these heuristic judges often yield high accuracy, they frequently lack formal guarantees concerning their reliability, reproducibility, and transparency. This paper introduces a comprehensive theoretical framework to evaluate post-hoc debate judges across formal properties like determinism, robustness, groundedness, and contestability. The authors then experimentally compare conventional LLM-based judges against computational argumentation semantics, specifically DF-QuAD, within claim verification tasks involving single-turn and multi-turn debate scenarios.

The study establishes an abstract setup where a set of agents produce opinion profiles containing stances and textual explanations. To systematically assess how judges process these profiles, the authors define several desirable formal properties. Determinism ensures identical inputs yield identical outputs. Permutation independence prevents agent ordering from biasing the results. Profile and judge robustness guard against minor perturbations in opinion or prompt variations. Non-hallucination and judge unanimity bind the final judgment closely to the participants' proposals. Finally, contestability requires that the judgment method responds logically and predictably when opinion profiles are modified to be more supportive of a claim.

Through experiments spanning single-turn isolated generation and multi-turn conversational refinement, the authors test variants where agents generate stances either before explanations, after explanations, or omit stances entirely. The results reveal that both LLM-as-a-judge and argumentation semantics achieve comparable accuracy performance across all tested scenarios. However, the argumentation-based judges significantly outperform LLM judges in satisfying the proposed formal properties—offering rigorous guarantees regarding robustness, monotonicity, and structured contestability that black-box language models fail to provide natively.