With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper about how we test AI agents—specifically, the software frameworks that help these agents actually get work done.
Sam: The puzzle is that while we often judge AI agents by whether they get the right answer, that tells us almost nothing about how they got there.
Alex: So "correctness" isn't enough to understand how these systems are actually performing?
Sam: Exactly. An AI agent doesn't work alone. It runs inside what researchers call a "harness"—think of it as the control room surrounding the AI. The harness manages the agent's instructions, its tools, and how it decides what to do next. The paper's central claim is that these harnesses aren't just passive wrappers. They fundamentally shape how an agent reasons, what it costs to run, and how often it makes mistakes.
Alex: So the same underlying AI model could behave quite differently depending on which harness it's sitting inside?
Sam: That's exactly what the study suggests. And here's the problem: if a developer wants to compare two different harnesses, they currently have almost no way to do that without building a custom testing setup for every single combination. It becomes an enormous engineering burden.
Alex: That's like needing a different power adapter for every appliance you own, instead of one universal plug.
Sam: That's a good way to put it. To solve this, the researchers built something called the Agent Task Protocol, or ATP. It's that universal plug—a standardised interface that lets any test or benchmark connect to any harness without needing custom code for every pairing.
Alex: So ATP removes the adapter problem. Once that connection is in place, what does the system actually do with it?
Sam: It deploys what the paper calls a "monitor"—a piece of software that watches the agent in real time and records everything it does. Every step the agent takes, every tool it calls, every moment of reasoning—all of it gets written into a structured log. You can think of it as a detailed diary of the agent's entire journey from the moment it receives a task to the moment it delivers an answer.
Alex: And that diary is what makes the evaluation possible?
Sam: Right. The paper calls this approach "Lifecycle-Aligned Evaluation." Instead of only looking at the final result, they break the agent's performance into distinct stages—reasoning, action, and answer—and measure quality at each one separately.
Alex: So you aren't just grading the final essay. You're grading the brainstorming, the research, and the drafting process as well. Why does that distinction matter so much?
Sam: Because two agents can arrive at the same correct answer through very different paths. One might have done it efficiently and safely. Another might have spent far more computational resources—which translates directly to money—or made several dangerous errors along the way that happened not to affect the final output. If you only look at the final score, those two agents look identical. The process metrics are what reveal the difference.
Alex: So the process metrics expose a kind of hidden quality that the final score completely obscures.
Sam: Precisely. And the study's findings suggest that no single harness is universally the best. Which harness performs well depends on the specific task and the specific model being used. That's a meaningful finding, because it means choosing a harness isn't a one-time decision—it requires ongoing, task-specific evaluation.
Alex: How does the system track all of this without turning into an unmanageable flood of data?
Sam: It relies on an existing standard called OpenTelemetry. Think of it like tracking a parcel through a global shipping network. Every time that parcel moves—from warehouse to truck to sorting facility—it gets scanned with a label that follows a consistent format, so anyone anywhere in the chain can read it. OpenTelemetry does the same thing for software events. It gives every action the AI takes a clear, standardised label, so the logs are readable and comparable across different systems.
Alex: So it turns what could be a chaotic stream of activity into a structured, legible record. How does it decide what counts as a meaningful action worth recording?
Sam: The framework uses a three-part structure. There's a "semantic layer" that gives each action a meaningful name—so the system knows whether the agent is searching the web or writing code, not just that something happened. There's a "span layer" that records when each action started and ended, so you can measure how long things took. And there's an "SDK layer" that handles the connection to whichever specific harness is being used. Together, these three layers act like a universal translator, converting the different languages spoken by different frameworks into one consistent, readable report.
Alex: It sounds like the deeper contribution here isn't just a new testing tool. It's a way to standardise how we observe the process of AI reasoning, not just the output.
Sam: That's a fair reading of it. By separating the evaluation from the framework being evaluated, A2E makes it possible to ask questions we couldn't ask before—not just "did the agent get it right?" but "how did it get there, what did it cost, and was the path safe?" For anyone building or deploying AI agents, those questions matter considerably more than a single correctness score.
Alex: That's a genuinely useful reframe. Thanks for walking through it, Sam—and thanks to everyone listening to ResearchPod.