Shun Zou, Ziyu Ma, Yi Zou, Yong Wang, Lin Chen, Zehui Chen, Guanhua Chen, Xiangxiang Chu, Feng Zhao
9 min
Abstract
Terminal agents are moving beyond local operations toward completing real-world professional tasks. Short trajectories can demonstrate commands and tool calls, but they do not fully show how agents accumulate evidence through exploration, use it to inform judgments, continually adjust strategies based on feedback, and turn intermediate results into final deliverables. High-quality long-horizon trajectories capture these agentic behavioral patterns and their connections within a single problem-solving process, providing richer experience for sustained autonomous work. However, environment synthesis may complete or simplify the key work underlying these patterns in advance and thus remove the need for exploration, implementation, and verification that the agent was meant to perform. To address this, we introduce TerminalHorizon, a fully automated data synthesis engine for long-horizon terminal agents. Starting from real-world professional work, it uses independent execution to identify key work omitted during construction and uses the resulting evidence to guide environment reconstruction. To extend these patterns across stages, it progressively expands tasks under the same goal, introducing new requirements that build on prior results and call for further exploration and improvement. Using this engine, we construct 1.5K task environments and perform supervised fine-tuning of Qwen3.5-27B on the resulting dataset to obtain TerminalHorizon-27B. We achieve substantial performance gains on all 12 benchmarks spanning coding-agent tasks, agentic tasks in real-world production settings, and general reasoning. Multiple RSI case studies, each with a 12-hour research budget, further demonstrate the model's ability to continue experimenting beyond an initial solution, improve its methods based on feedback, and retain effective improvements in its final deliverables.
Alex: Can you make that distinction concrete? Otherwise, “preserving substantive work” could mean whatever the task designer prefers.
Sam: Consider their radio-calibration case. The task asks an assistant to calibrate a synthetic radio observation while preserving raw measurements and metadata. But the initial environment includes tools that perform the numerical fitting and write calibration files.
Alex: Then the assistant can operate a calibration workflow without developing the calibration method. Passing the task does not show that it practiced the intended numerical work.
Sam: The independent attempt exposed that gap. Reconstruction removed the fitting routines but kept data inspection, calibration application, and feedback about remaining error. It also made the required data axes and file conventions public, so removing the shortcut did not turn the task into guesswork.
Alex: What happened when another assistant tried the reconstructed version?
Sam: It inspected the data, implemented a fitting workflow, and revised its parameters after seeing high residual errors. It then checked held-out observations and verified that the raw data remained unchanged. Both versions passed, but only the reconstructed version required the numerical modeling and implementation.
Alex: That repairs one task. How do they extend it without merely adding unrelated chores?
Sam: Their second stage adds requirements under the same goal. Those requirements must go beyond what the original solution can directly satisfy. They can deepen a mechanism, introduce richer evidence, or require a method upgrade, with later work depending on earlier observations, artifacts, or feedback.
Alex: There is also a fairness issue here. If the checker demands something hidden, apparent difficulty could just be an undisclosed requirement.
Sam: The authors explicitly prohibit that. Scoring requirements must follow from public instructions and discoverable evidence. Test data can stay private, but the checker cannot introduce a completion requirement the assistant had no way to discover.
Alex: Let’s turn to evidence. How much training data did this process produce, and what did they do with it?
Sam: They retained about three thousand training trajectories across a broad range of professional domains. They used supervised fine-tuning: training an existing model on those examples. The student was a Qwen model, and the resulting assistant is called TerminalHorizon.
Alex: Were the examples selected simply because they succeeded? That could remove the failures that make extended work realistic.
Sam: No. Their reviewer evaluates coherent interaction, progress grounded in environmental evidence, and useful connections across turns. An unsuccessful attempt can remain if it demonstrates sound diagnosis or recovery. Persistent ineffective repetition, leaked answers, and prohibited shortcuts are excluded.
Alex: What is the clearest performance comparison against the model before training?
Sam: On Terminal-Bench, a benchmark of terminal tasks, the trained model solved about fifty-six percent in the second version. The base model solved about forty-two percent. That comparison uses the same execution framework, and the authors repeat tasks and report average performance.
Alex: That is a substantial difference on terminal work. Does the evidence stop at the interface used for training?
Sam: The trained model improved across all twelve evaluated benchmarks. Those included enterprise automation, customer-service tasks, cross-domain service tasks, and general reasoning. The authors also report a closely matched comparison against Terminal-Universe, using the same base model and execution framework, but much less training data.
Alex: Fewer trajectories do not necessarily mean less training computation. And several things changed together here. Can we attribute the improvement specifically to longer examples?
Sam: Not from these comparisons. They support the complete training recipe, but do not isolate length from reconstruction, expansion, teacher behavior, or filtering. My reading is that the method is promising, while the contribution of each ingredient remains unresolved.
Alex: The paper also discusses autonomous research. What does that add beyond benchmark scores?
Sam: It compares the trained and base models on several research cases, each with a twelve-hour budget. In point-cloud fusion, the task is to reconstruct a complete scene from noisy, partially overlapping views. The base model stops early after obtaining a high score on its own proxy metric—an accessible substitute for the real objective.
Alex: But agreement with one view could reward copying that view, rather than recovering surfaces seen elsewhere.
Sam: That is the problem the authors identify. The trained model questions a similarly misleading measure, inspects which views contribute points, and tests coverage by leaving a view out. It revises alignment and point selection, and its final submission scores better on hidden scenes.
Alex: Those traces explain the behavior, but do they establish repeatable research improvement?
Sam: Each case has only one research run per model. The models can also stop voluntarily, so actual research time differs. These are informative demonstrations of diagnosis and revision, not a repeated estimate that separates better methods from greater persistence. Their checks of training-task descriptions found no evidence of benchmark reuse, which is reassuring but does not resolve those limitations.
Alex: Who should read the full document, and where should they begin?
Sam: Researchers building assistant training data should start with “Challenge-Guided Long-Horizon Synthesis,” then the reconstruction case studies. They show where apparently demanding tasks quietly remove the work. Researchers evaluating autonomous research should read the additional long-horizon applications alongside the evaluation protocols, especially the final-submission scoring and single-run design.
Alex: And what should everyone else carry away from it?
Sam: Longer practice matters when evidence changes the next decision, not when the transcript merely gets longer. This preprint offers an encouraging way to construct that practice, rather than assuming difficult-looking tasks already contain it.
Alex: Thanks for listening.