Zongxia Li, Zhongzhi Li, Yucheng Shi, Ruhan Wang, Junyao Yang, Zhichao Liu, Xiyang Wu, Anhao Li, Yue Yu, Ninghao Liu, Lichao Sun, Haotao Mi, LeoweiLiang
5 min
Abstract
AI agents have become capable of autonomously completing short, well-specified tasks. However, existing terminal benchmarks largely focus on simple problems that finish within minutes and are evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, yielding sparse reward signals and an incomplete picture of agent capability. We introduce Long-Horizon-Terminal-Bench, a terminal benchmark of 46 long-horizon tasks spanning nine categories, including experiment reproduction, software engineering, multimodal analysis, interactive games, and scientific computing. Each task follows a Terminal-Bench-style setup with a reference solution or simulation engine, but is further decomposed into fine-grained graded subtasks. This design enables dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on open-ended workflows. Tasks in Long-Horizon-Terminal-Bench typically require hundreds of episodes and minutes to hours of execution, stressing long-horizon planning, long-context management, and iterative debugging rather than one-shot problem solving. We evaluate 15 frontier models and find that agents consume on average 9.9M tokens per task, with roughly 231 episodes and 85.3 minutes of execution time per run, making Long-Horizon-Terminal-Bench more demanding than prior terminal-based benchmarks. Even the strongest tested model achieves 15.2% pass@1 at a partial-reward threshold of 0.95 and 10.9% at a perfect-reward threshold of 1.0, while the mean pass rate across models is 4.3% and 1.7% under the two thresholds, respectively. These results reveal headroom for improvement. We further analyze failure modes and error patterns, and release Long-Horizon-Terminal-Bench to support future progress on long-horizon terminal agents.
Alex: So a binary system is actually hiding useful information. We'd be treating "almost finished" the same as "completely lost."
Sam: Precisely. It's like looking at a student's final exam grade and concluding they learned nothing — even if they got every intermediate step right. The dense approach lets researchers see exactly where an agent stops making progress, which tells you precisely which part of its reasoning needs work.
Alex: And all of this happens inside some kind of controlled digital environment, right? So the AI isn't accidentally breaking anything on a real computer?
Sam: Right. They run everything inside what's called a "containerized environment" — think of it as a sealed practice room. The agent can do whatever it needs to do in there without affecting anything outside. It also makes the tests consistent and repeatable, so you can fairly compare different AI models against each other. And they include hidden stress tests — like intentionally leaving out data the agent might expect — to make sure it's actually solving the problem rather than just pattern-matching from examples it's seen before.
Alex: So it's testing for genuine understanding, not just memorisation.
Sam: That's the idea. And because the agent can only interact through a command-line interface — basically a text-only terminal, no graphical shortcuts — it has to truly understand what it's doing, step by step, the way a human engineer would.
Alex: What about the cost of running all this? These must be fairly expensive tests to run.
Sam: They are. Each task runs to a meaningful cost, and the researchers found something worth paying attention to: spending more on a model doesn't reliably produce better results. Some of the more expensive models actually performed worse than cheaper ones on these long-horizon tasks. So throwing more computing power at the problem is not a shortcut to reliability.
Alex: Which suggests the real issue isn't processing power at all.
Sam: Exactly. The agents that performed best weren't necessarily the most powerful in a general sense — they were the ones that could monitor their own progress and catch their own mistakes before moving on. The paper calls this "agent self-verification," and it's where most models currently fall short. Without it, an agent is essentially working through a complex task without ever pausing to check whether it's actually on track.
Alex: It's a bit like trying to assemble something complicated without ever consulting the instructions to see if you've missed a step.
Sam: That's a fair comparison. And it points to what the researchers see as the path forward: agents need better planning and more calibrated judgment about when they're actually done. Not just the ability to execute steps, but the ability to look back at their own work and recognise when something is missing before they call it finished.
Alex: So this benchmark isn't just measuring current performance — it's pointing at the specific capability that needs to be built next.
Sam: That's the broader contribution. By making the grading granular enough to see partial progress, the researchers have turned what used to be a simple pass-or-fail verdict into a detailed map of where AI agents succeed, where they stall, and what kind of reasoning they still lack. That's a meaningful step forward for the field.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.