You Don’t Need to Run Every Eval

Modern AI development involves running dozens of benchmarks for every model release, checkpoint, and design iteration. This process is both time-consuming and expensive. This paper investigates whether this exhaustive testing is necessary by analyzing a large-scale matrix of 84 frontier models across 133 benchmarks. The authors find that this matrix is remarkably sparse but highly structured, exhibiting an "effectively rank-2" geometry. This implies that the vast majority of variation in model performance across diverse tasks—from coding and math to reasoning and agentic capabilities—can be captured by just two underlying factors.

Building on this geometric insight, the authors introduce BenchPress, a matrix completion method designed to predict missing benchmark scores. By treating the score matrix as a logit-space rank-2 system, BenchPress can infer held-out scores with a median absolute error of approximately 4.6 points. The authors also provide a confidence layer that estimates the reliability of these predictions, allowing practitioners to decide when an inferred score is sufficient and when a full evaluation is required.

One of the most significant contributions of this work is the identification of compact "probe sets." The authors demonstrate that by evaluating a model on a small, carefully selected subset of just five benchmarks—such as GPQA-D, MMLU-Pro, and Codeforces—researchers can predict the model's performance on the rest of the scorecard with high fidelity. This approach allows for "budgeted scorecards," where developers can estimate a model's full profile without the overhead of running the entire suite. The authors have released an interactive tool and the underlying score matrix to facilitate this practice.