We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to evaluate AI coding assistants. Sam, what's the core problem this paper is trying to solve?
Sam: Most current benchmarks for coding AI are what the authors call "outcome-blind." They only check if the agent finished the task correctly—like a pass-or-fail grade on a test. But for a developer actually using one of these tools day-to-day, the process of getting there matters just as much as the final code.
Alex: So a simple "pass" doesn't tell the whole story?
Sam: Exactly. An agent might pass a final test but still be frustrating to work with—making messy, unrequested file edits, or ignoring your instructions along the way. The authors propose a new benchmark called AgentLens that shifts the focus from the final result to the entire "trajectory."
Alex: A "trajectory"—that's the whole history of the interaction?
Sam: That's right. It's the complete record of the session: every message from the user, every tool the agent called, and how it handled errors. Think of it like a driving test. Instead of just checking whether the car arrived at the destination, the examiner grades the driver on lane discipline, signal usage, and how they handled unexpected traffic.
Alex: That makes sense. But how do you turn that messy, back-and-forth interaction into an actual score? It sounds like you'd need a human watching every single session.
Sam: That's where the "LLM-as-judge" approach comes in. The system uses a large language model—another AI—to act as a reviewer. It reads the entire history of the session and produces a qualitative report on the agent's performance, flagging things like whether the agent followed instructions or used its tools correctly.
Alex: So it's automating the role of a human code reviewer?
Sam: Precisely. But it doesn't rely solely on the judge's opinion. It pairs those reviews with what the paper calls "formal verification"—objective, automated checks like running unit tests or inspecting file contents. By combining the two, you get a score backed by both hard data and a nuanced explanation.
Alex: What if the agent "hacks" the test? Like, finding a way to pass the formal check while doing something risky to get there?
Sam: That's exactly what the trajectory review is designed to catch. The researchers call this "reward hacking." Because the system reviews the entire path—not just the endpoint—it can flag if the agent used a dangerous shortcut or ignored the user's constraints, even if the final code technically works.
Alex: So it's not just about the destination. It's about the quality of the journey.
Sam: Exactly. They even test agents against different "user personas"—including a frustrated or difficult user—to see how the agent handles pressure. The idea is to check whether the AI is robust enough for real-world development, not just ideal conditions.
Alex: How do they pull all of that together into one final number?
Sam: They create what they call a "Quality Index." Think of it like a final grade in a class that averages your homework, quizzes, and exam scores. They take the average of the judge's metrics and the formal verification score to get a single proxy for how production-ready an agent is.
Alex: But if you just average everything, don't you lose the nuance of why an agent failed?
Sam: That's why they don't stop at the index. They also look at what they call "metric profiles"—the individual scores laid out side by side. They found that some agents trade off interaction quality for raw speed, for instance. The index gives you a quick snapshot, but the individual metrics are where the real diagnostic work happens.
Alex: Wait—if you're using an AI to judge another AI, doesn't it risk favouring models from its own "family"?
Sam: That is a genuine concern, and the researchers acknowledge it. They found that in roughly one in five cases, different judge models disagreed with each other—mostly on the more subjective metrics, like how pleasant the agent's responses felt. The paper suggests using a single, fixed judge model to keep comparisons fair across the leaderboard.
Alex: And what about the formal verification side? Is that more stable?
Sam: Somewhat, but not perfectly. The researchers found that variance in those scores wasn't random noise—it reflected the agent genuinely struggling with specific, difficult scenarios. So the instability is actually informative. It tells you exactly where the agent is fragile, rather than just showing a score that dipped for no clear reason.
Alex: So the inconsistency is a feature, not a flaw. It points you to the exact weak spots.
Sam: Precisely. By identifying those unreliable tasks, developers can target the exact scenarios where the agent breaks down, rather than guessing why the overall score dropped.
Alex: What are the broader limitations here? Using an AI to judge another AI can't be perfect.
Sam: The authors are candid about this. The validity of the LLM-based reviews currently rests on internal experience with human annotators, rather than a formal study measuring how often a human and an AI judge actually agree on complex sessions. Quantifying that agreement is, as they put it, a critical next step.
Alex: Is the benchmark limited to one type of coding?
Sam: For now, it's focused on Java. But the design is intentionally modular—developers can connect different coding agents by implementing a standard interface. The plan is to grow the benchmark by adding focused "task families," like refactoring or debugging, rather than lumping all coding work into one undifferentiated pile.
Alex: Are there other factors that could skew results?
Sam: Hardware and serving conditions are a real issue. If one agent runs on faster servers, it might complete more work within a fixed time limit simply because it's quicker, not because it's more capable. Separating raw model quality from infrastructure speed is a persistent challenge the paper flags but doesn't fully resolve.
Alex: It sounds like the deeper ambition here is to make AI evaluation more rigorous—less of a popularity contest and more of a genuine diagnostic.
Sam: That's exactly how the authors frame it. By providing an evidence-linked review for every score, AgentLens aims to move the field toward evaluation that actually helps developers understand what their tools can and can't do. It's a meaningful step toward building agents that are ready for real production environments, not just benchmark leaderboards.
Alex: Thanks for walking us through this, Sam. And thanks for listening to ResearchPod.