AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to test how well AI agents can handle long, complicated projects.
Sam: The paper we're discussing introduces a testing framework called Long-Horizon-Terminal-Bench. The central claim is that current AI agents are quite capable at short, quick tasks — but they struggle to maintain focus and accuracy over the kind of long, multi-step workflows that human experts actually perform every day.
Alex: So the paper is basically asking: why do AI agents seem to "get lost" when they have to work on a project for an hour instead of a minute?
Sam: That's exactly the puzzle. Most current tests for AI agents are like short sprints. They ask the agent to fix one line of code or run a single command. And then it's graded on a simple pass-or-fail basis — either the job is done, or it isn't. Researchers call this a "binary" outcome.
Alex: That sounds like a real limitation. If I'm writing a long essay and I nail the introduction and three body paragraphs but stumble on the conclusion, I shouldn't get a zero for the whole thing.
Sam: You've hit on the core issue. In the real world, complex work is a series of smaller milestones. If an agent fails at the very last step of a two-hour task, a binary test tells us nothing about whether it did the first ninety percent correctly — or whether it was failing from the very start. This research replaces that all-or-nothing approach with what they call "dense" grading.
Alex: So instead of just checking the final result, they're evaluating the agent's work at every stage of the process?
Sam: Exactly. Think of it like a maths exam where the teacher gives partial credit for the steps you took to reach your answer. The researchers break a long, multi-hour task — like reproducing a climate study or setting up a complex software environment — into a sequence of smaller, meaningful goals. Each of those goals has its own automated check. It's essentially a progress bar for the agent's reasoning.
Alex: That's a much more useful picture. But how do these agents actually perform when you put them through these longer, more realistic tests?
Sam: The results are sobering. Even the most capable models struggle to cross the finish line on these extended tasks. The bottleneck isn't just raw intelligence — it's the ability to sustain a coherent plan over time. And here's where the dense grading becomes genuinely useful: nearly twice as many runs were "near-complete" as were "fully successful." Without that granular scoring, all those near-misses would just disappear into the same failure bucket as a total crash.
Alex: So a binary system is actually hiding useful information. We'd be treating "almost finished" the same as "completely lost."
Sam: Precisely. It's like looking at a student's final exam grade and concluding they learned nothing — even if they got every intermediate step right. The dense approach lets researchers see exactly where an agent stops making progress, which tells you precisely which part of its reasoning needs work.
Alex: And all of this happens inside some kind of controlled digital environment, right? So the AI isn't accidentally breaking anything on a real computer?
Sam: Right. They run everything inside what's called a "containerized environment" — think of it as a sealed practice room. The agent can do whatever it needs to do in there without affecting anything outside. It also makes the tests consistent and repeatable, so you can fairly compare different AI models against each other. And they include hidden stress tests — like intentionally leaving out data the agent might expect — to make sure it's actually solving the problem rather than just pattern-matching from examples it's seen before.
Alex: So it's testing for genuine understanding, not just memorisation.
Sam: That's the idea. And because the agent can only interact through a command-line interface — basically a text-only terminal, no graphical shortcuts — it has to truly understand what it's doing, step by step, the way a human engineer would.
Alex: What about the cost of running all this? These must be fairly expensive tests to run.
Sam: They are. Each task runs to a meaningful cost, and the researchers found something worth paying attention to: spending more on a model doesn't reliably produce better results. Some of the more expensive models actually performed worse than cheaper ones on these long-horizon tasks. So throwing more computing power at the problem is not a shortcut to reliability.
Alex: Which suggests the real issue isn't processing power at all.
Sam: Exactly. The agents that performed best weren't necessarily the most powerful in a general sense — they were the ones that could monitor their own progress and catch their own mistakes before moving on. The paper calls this "agent self-verification," and it's where most models currently fall short. Without it, an agent is essentially working through a complex task without ever pausing to check whether it's actually on track.
Alex: It's a bit like trying to assemble something complicated without ever consulting the instructions to see if you've missed a step.
Sam: That's a fair comparison. And it points to what the researchers see as the path forward: agents need better planning and more calibrated judgment about when they're actually done. Not just the ability to execute steps, but the ability to look back at their own work and recognise when something is missing before they call it finished.
Alex: So this benchmark isn't just measuring current performance — it's pointing at the specific capability that needs to be built next.
Sam: That's the broader contribution. By making the grading granular enough to see partial progress, the researchers have turned what used to be a simple pass-or-fail verdict into a detailed map of where AI agents succeed, where they stall, and what kind of reasoning they still lack. That's a meaningful step forward for the field.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.