Andrey Podivilov, Vadim Lomshakov, Sergey Savin, Matvei Startsev, Roman Pozharskiy, Maksim Parshin, Sergey Nikolenko
6 min
Abstract
We present AgentLens, a production-assessed benchmark for interactive code agents. Most code-agent benchmarks reduce a run to a single bit -- did the task pass? -- but the people who actually use these agents experience the entire trajectory: how the agent follows instructions, uses its tools, verifies its own work, recovers from mistakes, and talks to them along the way. AgentLens evaluates that whole trajectory. It pairs formal verification, where an objective check exists, with LLM-written trajectory reviews and side-by-side comparisons, so that each run yields a readable explanation of why the score is what it is. This makes AgentLens useful for more than ranking models: we use it to diagnose model behavior, compare successive versions of our own agent, and catch product regressions in a nightly evaluation pipeline. We release the benchmark as open source at https://github.com/agent-lens/agent-lens-bench.
Sam: Exactly. They even test agents against different "user personas"—including a frustrated or difficult user—to see how the agent handles pressure. The idea is to check whether the AI is robust enough for real-world development, not just ideal conditions.
Alex: How do they pull all of that together into one final number?
Sam: They create what they call a "Quality Index." Think of it like a final grade in a class that averages your homework, quizzes, and exam scores. They take the average of the judge's metrics and the formal verification score to get a single proxy for how production-ready an agent is.
Alex: But if you just average everything, don't you lose the nuance of why an agent failed?
Sam: That's why they don't stop at the index. They also look at what they call "metric profiles"—the individual scores laid out side by side. They found that some agents trade off interaction quality for raw speed, for instance. The index gives you a quick snapshot, but the individual metrics are where the real diagnostic work happens.
Alex: Wait—if you're using an AI to judge another AI, doesn't it risk favouring models from its own "family"?
Sam: That is a genuine concern, and the researchers acknowledge it. They found that in roughly one in five cases, different judge models disagreed with each other—mostly on the more subjective metrics, like how pleasant the agent's responses felt. The paper suggests using a single, fixed judge model to keep comparisons fair across the leaderboard.
Alex: And what about the formal verification side? Is that more stable?
Sam: Somewhat, but not perfectly. The researchers found that variance in those scores wasn't random noise—it reflected the agent genuinely struggling with specific, difficult scenarios. So the instability is actually informative. It tells you exactly where the agent is fragile, rather than just showing a score that dipped for no clear reason.
Alex: So the inconsistency is a feature, not a flaw. It points you to the exact weak spots.
Sam: Precisely. By identifying those unreliable tasks, developers can target the exact scenarios where the agent breaks down, rather than guessing why the overall score dropped.
Alex: What are the broader limitations here? Using an AI to judge another AI can't be perfect.
Sam: The authors are candid about this. The validity of the LLM-based reviews currently rests on internal experience with human annotators, rather than a formal study measuring how often a human and an AI judge actually agree on complex sessions. Quantifying that agreement is, as they put it, a critical next step.
Alex: Is the benchmark limited to one type of coding?
Sam: For now, it's focused on Java. But the design is intentionally modular—developers can connect different coding agents by implementing a standard interface. The plan is to grow the benchmark by adding focused "task families," like refactoring or debugging, rather than lumping all coding work into one undifferentiated pile.
Alex: Are there other factors that could skew results?
Sam: Hardware and serving conditions are a real issue. If one agent runs on faster servers, it might complete more work within a fixed time limit simply because it's quicker, not because it's more capable. Separating raw model quality from infrastructure speed is a persistent challenge the paper flags but doesn't fully resolve.
Alex: It sounds like the deeper ambition here is to make AI evaluation more rigorous—less of a popularity contest and more of a genuine diagnostic.
Sam: That's exactly how the authors frame it. By providing an evidence-linked review for every score, AgentLens aims to move the field toward evaluation that actually helps developers understand what their tools can and can't do. It's a meaningful step toward building agents that are ready for real production environments, not just benchmark leaderboards.
Alex: Thanks for walking us through this, Sam. And thanks for listening to ResearchPod.