Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

Overview

Evaluating AI agents is currently fragmented, with benchmarks using inconsistent tasks, verifiers, and scoring rules. This makes it difficult to compare agent performance across the field or track genuine progress. The authors introduce Messier, a unified corpus containing 957,253 records from 30 benchmarks, 714 agents, and 11,891 tasks. By standardizing these records—including model, scaffold, environment, and verifier information—the authors provide a foundational infrastructure for auditing agent capabilities and analyzing evaluation failures.

To build Messier, the authors consolidated 24 existing public benchmark releases and added new, uniform evaluations across six underrepresented professional and scientific domains, such as legal review and quantum algorithm design. The dataset is organized into five functional groups: programming, research and reasoning, enterprise workflows, GUI navigation, and function calling. Each record is mapped to a standardized data model and annotated with SOC (occupational) and NAICS (industry) classifications. This allows for fine-grained analysis that goes beyond simple aggregate pass rates, enabling researchers to perform counterfactual rescoring to see how different aggregation rules affect agent rankings.

Findings

Using this corpus, the authors demonstrate that frontier progress is highly uneven. Function calling has reached a point of saturation, and programming tasks show the fastest improvement. In contrast, enterprise workflows remain the most challenging category for current agents. Furthermore, the authors show that "all-pass" aggregation rules—where every verifier must succeed for a task to be marked as passed—can obscure performance gains and artificially alter agent rankings, suggesting that headline metrics often hide significant nuances in agent behavior.

Messier reduces the need for costly, redundant evaluation runs by providing a shared, inspectable dataset. By enabling researchers to slice performance data by occupation, action space, or verifier type, it allows for more rigorous auditing of agent capabilities. The corpus also provides a template for future evaluations, helping the community move toward more comparable and transparent benchmarking standards.