Evaluating LLM agents on benchmarks like SWE-Bench and GAIA can be expensive, time-consuming, and requires complex infrastructure. A single evaluation can cost thousands of dollars and take days to complete. In contrast, non-agentic LLM benchmarks that test individual capabilities (e.g., reasoning, code generation) are fast and cheap to run. In this paper, we investigate whether performance on expensive agentic benchmarks can be accurately predicted by the performance on a small, carefully selected subset of atomic evaluation instances. We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks. Given a pool of candidate instances spanning atomic capabilities, PACE fits a regression that maps a model's scores on a compact subset of source instances to its score on the target agentic benchmark. The subset itself is curated by combining two complementary instance-selection strategies, target-relevance local selection and globally informative global selection. We apply PACE to the 4 target agentic benchmarks in this paper, which yields PACE-Bench, the concrete proxy benchmark that we evaluate in the paper. Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost. We further analyze the selected proxy instances, revealing which skills each agentic benchmark uniquely demands. PACE enables practitioners to obtain reliable estimates of agentic performance during model development, selection, and routing, without the overhead of full agent evaluation.
Alex: Welcome to another episode of ResearchPod. Today we're looking at PACE, a framework designed to solve a significant bottleneck in how we evaluate the most capable AI models.
Sam: I've heard testing these AI agents is incredibly expensive. Is this paper essentially trying to find a faster way to measure performance?
Alex: That's exactly the problem it's addressing. Imagine you've built a new AI model and you want to know how well it performs on difficult, real-world tasks—things like browsing the web, writing and running code, or completing a multi-step research project. These tests, called "agentic" benchmarks, can take days to run and cost enormous amounts of computing power.
Sam: So if you're a developer tweaking your model dozens of times a week, you simply can't afford to run the full test every time.
Alex: Precisely. You'd be stuck waiting days for feedback on every small change. PACE is designed to break that bottleneck. It selects a small, carefully chosen set of much simpler tasks—things like basic math problems or short coding exercises—that act as a kind of diagnostic panel. The idea is that a model's performance on these quick, cheap tasks can reliably predict how it would do on the expensive, full-scale test.
Sam: Like a doctor using a few blood markers to estimate your overall health, rather than running every possible scan.
Alex: That's a good way to put it. And the key challenge is choosing the right blood markers—the tasks that are genuinely informative, not just easy to measure.
Sam: So how does PACE actually pick those tasks?
Alex: It uses two signals working together. The first is about relevance: PACE looks for tasks that are mathematically similar to the expensive agentic test it's trying to predict. You want your cheap diagnostic to be related to the real thing, not just any random question.
Sam: That makes sense. What's the second signal?
Alex: The second is about diversity. Imagine you have a hundred practice questions, but fifty of them are basically asking the same thing in slightly different words. Including all fifty doesn't give you fifty times the information—it's mostly repetition. PACE uses a technique called Singular Value Decomposition to cut through that redundancy. Think of it as mapping out the "shape" of a collection of questions. It identifies which questions are genuinely distinct from each other, so you end up with a small set that covers as much ground as possible.
Sam: So you're not just picking the most relevant tasks—you're picking a diverse set of relevant tasks. That's a meaningful distinction.
Alex: Exactly. Once PACE has that curated subset, it fits what's called a regression—essentially a statistical line drawn through past data. It says: here's how a range of models scored on these cheap tasks, and here's how those same models scored on the expensive benchmark. That relationship becomes the prediction engine.
Sam: And I'd imagine model performance isn't perfectly consistent—there's some randomness involved?
Alex: There is, and the researchers account for it. They use a technique called bootstrapping, where the system repeatedly resamples its data in slightly different ways to test whether the prediction holds up. It's a way of stress-testing the result to make sure it isn't just a lucky coincidence.
Sam: So does it actually work? Can a cheap test genuinely stand in for a days-long evaluation?
Alex: The results are notable. Across fourteen models, PACE predicted agentic benchmark scores with high accuracy, while costing less than one percent of what a full evaluation would require. That's a substantial efficiency gain for developers who need rapid feedback during the development process.
Sam: If you can get a reliable estimate in minutes instead of days, it genuinely changes how you work. But I want to push back a little—if the task selection is that aggressive, doesn't it risk becoming too narrow? What if the cheap tasks you've chosen just happen to correlate with the benchmark for the wrong reasons?
Alex: That's a fair challenge, and the authors take it seriously. They address it in two ways. First, they draw from a deliberately broad pool of source tasks, so the selection isn't constrained to one narrow domain. Second, they use a learned weighting system that prevents any single question type from having an outsized influence on the prediction. The goal is to capture underlying capabilities—the fundamental skills that drive performance—rather than surface-level patterns that might not generalize.
Sam: So the logic is that if a model has the right foundational skills, those will show up across many different types of simple tasks, and PACE is designed to detect that signal.
Alex: That's the core idea. It's less about any specific question and more about identifying the building blocks that predict success on complex, multi-step goals.
Sam: It's a practical answer to a real resource problem. Rather than treating every evaluation as a full-scale production, PACE gives developers a reliable early read—with the understanding that the full test still matters when it counts.
Alex: That's a good summary. It's a tool for rapid iteration, not a final verdict. The full-scale validation still has its place. Thanks for listening to ResearchPod.