Haoning Wang, Mingxun Zhang, Chenyue Yu, Yingjun Shang, Xia Hu, Guanchu Wang, Na Zou
5 min
Abstract
With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.
Sam: Because two agents can arrive at the same correct answer through very different paths. One might have done it efficiently and safely. Another might have spent far more computational resources—which translates directly to money—or made several dangerous errors along the way that happened not to affect the final output. If you only look at the final score, those two agents look identical. The process metrics are what reveal the difference.
Alex: So the process metrics expose a kind of hidden quality that the final score completely obscures.
Sam: Precisely. And the study's findings suggest that no single harness is universally the best. Which harness performs well depends on the specific task and the specific model being used. That's a meaningful finding, because it means choosing a harness isn't a one-time decision—it requires ongoing, task-specific evaluation.
Alex: How does the system track all of this without turning into an unmanageable flood of data?
Sam: It relies on an existing standard called OpenTelemetry. Think of it like tracking a parcel through a global shipping network. Every time that parcel moves—from warehouse to truck to sorting facility—it gets scanned with a label that follows a consistent format, so anyone anywhere in the chain can read it. OpenTelemetry does the same thing for software events. It gives every action the AI takes a clear, standardised label, so the logs are readable and comparable across different systems.
Alex: So it turns what could be a chaotic stream of activity into a structured, legible record. How does it decide what counts as a meaningful action worth recording?
Sam: The framework uses a three-part structure. There's a "semantic layer" that gives each action a meaningful name—so the system knows whether the agent is searching the web or writing code, not just that something happened. There's a "span layer" that records when each action started and ended, so you can measure how long things took. And there's an "SDK layer" that handles the connection to whichever specific harness is being used. Together, these three layers act like a universal translator, converting the different languages spoken by different frameworks into one consistent, readable report.
Alex: It sounds like the deeper contribution here isn't just a new testing tool. It's a way to standardise how we observe the process of AI reasoning, not just the output.
Sam: That's a fair reading of it. By separating the evaluation from the framework being evaluated, A2E makes it possible to ask questions we couldn't ask before—not just "did the agent get it right?" but "how did it get there, what did it cost, and was the path safe?" For anyone building or deploying AI agents, those questions matter considerably more than a single correctness score.
Alex: That's a genuinely useful reframe. Thanks for walking through it, Sam—and thanks to everyone listening to ResearchPod.