Does Multi-Agent Debate Improve AI Feedback on Research Papers?

As AI tools for research assistance proliferate, a key debate has emerged: does increasing 'test-time compute'—by chaining multiple agents to debate and critique one another—actually produce better feedback than a single, high-quality model pass? The authors investigated whether multi-agent debate systems, which are designed to catch errors through adversarial cross-examination, provide more useful feedback to researchers than a standard single-pass AI review.

The researchers conducted a pre-registered, identity-masked experiment involving 44 economics meta-analyses. For each paper, they generated three types of feedback: a single-pass report from a frontier model, a report from a two-model adversarial audit (mad-research), and a report from a multi-agent workshop (paper-workshop). To ensure a fair comparison, all reports were normalized to a common length and template, and the authors of the papers ranked the reports by usefulness without knowing which tool produced which output.

Contrary to the authors' expectations, the single-pass configuration was preferred over both multi-agent tools. The authors ranked the single-pass report ahead of the mad-research tool on 32 of 44 papers and ahead of the paper-workshop tool on 30 of 44 papers. The study also found a significant disconnect between human and AI judges: while the authors valued their original human journal referee reports highly, AI judges consistently ranked those same human reports last. Furthermore, when an external AI model (Gemini) was asked to rank the AI-generated reports, it preferred the more complex multi-agent tools, directly contradicting the preferences of the human authors. This suggests that relying on AI to judge the quality of AI feedback may lead to misleading conclusions.

This study provides a critical reality check for the development of AI research assistants. It suggests that for the specific task of providing feedback on finished drafts, the marginal utility of increased computational complexity is low. The findings highlight the importance of evaluating AI tools through the lens of the intended end-user rather than relying on automated benchmarks or AI-as-a-judge protocols, which may prioritize different features than human researchers.