Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng, Yunze Xiao, Daniel Lee, Aditya Bharat Soni, Vincent Lo, Xiang Yue, Graham Neubig
5 min
Evaluating Large Language Model (LLM) agents on benchmarks like SWE-Bench or GAIA is notoriously difficult. These tasks require models to operate over long horizons, interact with complex software environments, and recover from errors, often resulting in evaluation processes that take days and cost thousands of dollars. This high barrier to entry limits how frequently researchers can assess progress, often restricting rigorous evaluation to only the most well-resourced teams.
This paper introduces PACE (Proxy for Agentic Capability Evaluation), a framework designed to predict agentic performance using a compact, inexpensive subset of non-agentic benchmark instances. Instead of running full agentic evaluations, PACE selects a small number of instances from existing, fast-to-run benchmarks—such as those measuring instruction following, code generation, or reasoning—that are most predictive of the target agentic benchmark.
PACE constructs this proxy by combining two selection strategies:
These selected instances are then used to fit a noise-aware regression model that maps a model's performance on the proxy subset to its expected performance on the full agentic benchmark.
Experiments across 14 models and 4 agentic benchmarks demonstrate that PACE is highly effective. Using a proxy of just 100 instances, PACE achieves a mean absolute error (MAE) under 4% and a Spearman correlation above 0.80, while reducing evaluation costs by approximately 100 times compared to random sub-sampling of the target benchmark. The framework also provides interpretable insights, as the selected proxy instances reveal the specific capabilities—such as planning or long-context aggregation—that are most critical for success on different agentic tasks.
Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.
Sam: And I'd imagine model performance isn't perfectly consistent—there's some randomness involved?
Alex: There is, and the researchers account for it. They use a technique called bootstrapping, where the system repeatedly resamples its data in slightly different ways to test whether the prediction holds up. It's a way of stress-testing the result to make sure it isn't just a lucky coincidence.
Sam: So does it actually work? Can a cheap test genuinely stand in for a days-long evaluation?
Alex: The results are notable. Across fourteen models, PACE predicted agentic benchmark scores with high accuracy, while costing less than one percent of what a full evaluation would require. That's a substantial efficiency gain for developers who need rapid feedback during the development process.
Sam: If you can get a reliable estimate in minutes instead of days, it genuinely changes how you work. But I want to push back a little—if the task selection is that aggressive, doesn't it risk becoming too narrow? What if the cheap tasks you've chosen just happen to correlate with the benchmark for the wrong reasons?
Alex: That's a fair challenge, and the authors take it seriously. They address it in two ways. First, they draw from a deliberately broad pool of source tasks, so the selection isn't constrained to one narrow domain. Second, they use a learned weighting system that prevents any single question type from having an outsized influence on the prediction. The goal is to capture underlying capabilities—the fundamental skills that drive performance—rather than surface-level patterns that might not generalize.
Sam: So the logic is that if a model has the right foundational skills, those will show up across many different types of simple tasks, and PACE is designed to detect that signal.
Alex: That's the core idea. It's less about any specific question and more about identifying the building blocks that predict success on complex, multi-step goals.
Sam: It's a practical answer to a real resource problem. Rather than treating every evaluation as a full-scale production, PACE gives developers a reliable early read—with the understanding that the full test still matters when it counts.
Alex: That's a good summary. It's a tool for rapid iteration, not a final verdict. The full-scale validation still has its place. Thanks for listening to ResearchPod.