Benchmarking the Benchmarks: Evaluating Benchmarks for Conversational Agents

As task-oriented conversational agents become more complex, the benchmarks used to evaluate them must be rigorous and reliable. However, both manually curated and automatically generated benchmarks often suffer from hidden flaws, such as inconsistent tasks, simplistic scenarios, or poor coverage of domain policies. This paper addresses the lack of systematic quality assessment for these benchmarks by proposing a reference-free evaluation framework.

The authors introduce four metrics based on LLM judges to evaluate benchmark quality without requiring a ground-truth reference dataset. These metrics assess:

• Consistency: Whether task descriptions and expected behaviors align with each other and with the domain policy.
• Complexity: The frequency of policy-violating scenarios, which are harder for agents to handle than standard compliant tasks.
• Coverage: The breadth of policy items tested across the entire benchmark.

To validate these metrics, the authors generated benchmarks using LLMs of varying capabilities (ranging from Llama-1B to GPT-5.4) and applied controlled perturbations—such as swapping expected behaviors or mismatched domain policies—to see if the metrics correctly identified the resulting quality degradation. They also compared the LLM-judge scores against independent human annotations.

The proposed metrics consistently distinguish between benchmark quality levels across different domains and judge models. The framework demonstrated a perfect or near-perfect ability to rank benchmarks generated by models of varying capabilities. Furthermore, the metrics proved highly sensitive to controlled quality-degrading perturbations, showing a clear decline in scores when tasks were corrupted. Human validation confirmed that the LLM-based scores align with human judgment, with moderate-to-strong statistical correlations across all metrics.