Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks

Research Overview

As frontier AI models become more capable, the benchmarks used to evaluate them have grown increasingly complex. This complexity introduces subtle failure modes—such as models accessing ground truth data or failing due to broken tools—that can invalidate performance claims. While manual auditing is the gold standard for detecting these flaws, it is prohibitively expensive and time-consuming at scale. This paper investigates whether automated transcript analysis, using AI-driven scanners, can effectively identify these benchmark quality issues.

The researchers developed a framework for automated auditing based on the Agentic Benchmark Checklist (ABC). They focused on four specific validity criteria: ground truth access, tool failure, guessing vulnerability, and answer format ambiguity. Using the Inspect Scout library, they built and iteratively refined scanners on a development set of transcripts, supplementing natural data with synthetic violations to ensure the scanners could recognize diverse failure patterns. The final scanners were then validated against human expert labels on a held-out test set of transcripts across several widely used agentic benchmarks, including SWE-Bench-Verified and CORE-Bench.

Findings and Implications

The study demonstrates that automated scanners can successfully identify verified quality issues in benchmarks that are often missed by random manual inspection. The authors provide a methodology for determining the necessary sample sizes to confidently assess benchmark validity, offering a path toward more systematic quality assurance. However, the results also reveal significant challenges: scanner performance is inconsistent across different benchmarks and criteria, and the reliance on specific model configurations for scanning suggests that these tools are not yet "plug-and-play." The authors conclude that while automated transcript analysis is a promising proof-of-concept for scaling benchmark audits, significant work remains to standardize evaluation reporting and improve scanner generalization.