Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig
5 min
Abstract
Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.
Sam: And I'd imagine model performance isn't perfectly consistent—there's some randomness involved?
Alex: There is, and the researchers account for it. They use a technique called bootstrapping, where the system repeatedly resamples its data in slightly different ways to test whether the prediction holds up. It's a way of stress-testing the result to make sure it isn't just a lucky coincidence.
Sam: So does it actually work? Can a cheap test genuinely stand in for a days-long evaluation?
Alex: The results are notable. Across fourteen models, PACE predicted agentic benchmark scores with high accuracy, while costing less than one percent of what a full evaluation would require. That's a substantial efficiency gain for developers who need rapid feedback during the development process.
Sam: If you can get a reliable estimate in minutes instead of days, it genuinely changes how you work. But I want to push back a little—if the task selection is that aggressive, doesn't it risk becoming too narrow? What if the cheap tasks you've chosen just happen to correlate with the benchmark for the wrong reasons?
Alex: That's a fair challenge, and the authors take it seriously. They address it in two ways. First, they draw from a deliberately broad pool of source tasks, so the selection isn't constrained to one narrow domain. Second, they use a learned weighting system that prevents any single question type from having an outsized influence on the prediction. The goal is to capture underlying capabilities—the fundamental skills that drive performance—rather than surface-level patterns that might not generalize.
Sam: So the logic is that if a model has the right foundational skills, those will show up across many different types of simple tasks, and PACE is designed to detect that signal.
Alex: That's the core idea. It's less about any specific question and more about identifying the building blocks that predict success on complex, multi-step goals.
Sam: It's a practical answer to a real resource problem. Rather than treating every evaluation as a full-scale production, PACE gives developers a reliable early read—with the understanding that the full test still matters when it counts.
Alex: That's a good summary. It's a tool for rapid iteration, not a final verdict. The full-scale validation still has its place. Thanks for listening to ResearchPod.