LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $τ^τ$-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $τ^τ$-bench to turn the work of cooperative agent building into a measurable target for coding agents.
Alex: Welcome to another episode of ResearchPod. Today we're looking at τ-bench — a benchmark designed to test whether AI systems can actually build software agents, rather than just act as them.
Sam: That distinction matters more than it might sound. We've gotten reasonably good at training models to perform as agents — to answer questions, call tools, complete tasks. What we haven't cracked is the engineering discipline of constructing those agents in the first place. This paper's argument is that current systems treat agent-building as a prompt-writing exercise, when it's actually a software engineering problem with all the messiness that implies.
Alex: So the benchmark is asking whether an AI can function as a junior developer — handed a noisy API spec, incomplete documentation, maybe some contradictory requirements — and deliver something that actually works under production conditions?
Sam: Exactly. And critically, something that works cost-effectively. The builder isn't just evaluated on whether the agent runs — it's evaluated on whether it passes a held-out stress test within a credit budget. That second constraint is where things get interesting.
Alex: What does the evaluation setup actually look like?
Sam: The key design choice is what they call sealed evaluation. The builder constructs the agent during a development phase, then that agent is frozen and deployed against test traffic the builder never saw. Construction and evaluation are fully decoupled. That matters because it closes off a class of shortcuts — you can't tune your system to the test cases if you don't know what they are. The agent has to be robust by design, not by memorization.
Alex: And how do current systems perform under that setup?
Sam: Poorly, and in a specific way. The strongest configuration they tested passed around 24% of tasks. Human experts, given the same setup, hit roughly 82%. That's not a small gap — it's the difference between a prototype and a deployable system. But the more informative finding is where the failures come from, because it's not primarily a capability gap in the coding sense.
Alex: So what is it?
Sam: It's process. The agents consistently fail at requirements elicitation. They stop gathering information too early, relying on shallow keyword searches through the documentation rather than actually interrogating the simulated client. They rarely ask clarifying questions. And then they ship the first design that compiles without validating it against anything resembling ground truth. It's the engineering lifecycle that breaks down, not the ability to write code.
Alex: That's a meaningful distinction. A model that can write correct code but skips the requirements phase is going to build the wrong thing correctly.
Sam: Right. And the unit economics constraint makes this worse, because it forces a real architectural decision. Without a budget, the naive move is to route everything through the most capable — and most expensive — model available. With a strict credit limit, the builder has to think about which subtasks actually need heavy compute and which can be handled cheaply. Current agents treat that as an afterthought. They don't develop a routing strategy; they just hit the budget wall and fail.
Alex: Does the benchmark do anything to prevent the agent from overfitting to whatever patterns exist in the development phase?
Sam: The authors address this through what they call task levers — parameters that vary API fidelity, workspace complexity, and the degree of ambiguity in the requirements. The idea is to force generalization rather than pattern matching. If the agent has only learned to handle clean, well-specified inputs, varying those levers will expose it. Whether that's sufficient to fully prevent overfitting is a fair question — it's more of a structural pressure than a formal guarantee — but it's a reasonable design choice given the difficulty of the problem.
Alex: Where would a careful reviewer push back on this?
Sam: A few places. The human expert ceiling of 82% is the benchmark's validity anchor — if that number is inflated, or if the experts had affordances the AI builders didn't, the gap looks worse than it is. The paper would need to be transparent about exactly what the human condition looked like. There's also a question about ecological validity: the tasks are synthetic by construction, and how well performance on τ-bench predicts performance on real-world agent-building workflows is something the authors can't fully establish from within the benchmark itself. That's not a fatal flaw — it's the standard limitation of any benchmark — but it means the 24% number should be read as a relative measure of current capability, not an absolute prediction of deployment readiness.
Alex: So the deeper claim here is that we need to treat agent construction as a measurable engineering discipline in its own right.
Sam: That's the frame the authors are pushing for. Right now, most evaluation infrastructure is built around what an agent does — its task completion rate, its tool use, its reasoning traces. τ-bench is asking us to evaluate what an agent builds, which requires a different kind of test. The sealed evaluation, the credit budget, the noisy requirements — those aren't arbitrary design choices. They're trying to operationalize the things that actually make agent deployment hard: ambiguity, resource constraints, and the need to generalize to traffic you haven't seen.
Alex: And the current results suggest we're quite far from systems that can handle that reliably.
Sam: A 24% pass rate against an 82% human ceiling, with failures concentrated in requirements gathering and validation — yes, there's substantial ground to cover. The interesting research question the benchmark opens up is whether the bottleneck is fundamentally about process — can you train or prompt a model to follow an engineering workflow — or whether it's something deeper about how these systems represent and reason about underspecified problems. τ-bench doesn't answer that, but it gives you the measurement apparatus to start asking it properly.
Alex: That's a useful place to leave it. Thanks for listening to ResearchPod.