Fanzhe Meng, Guoxin Chen, Jiale Zhao, Shuang Sun, Zhiyu Lin, Wayne Xin Zhao, Ruihua Song, Ji-Rong Wen, Kai Jia
6 min
As large language model agents increasingly rely on terminal interfaces to perform complex computing tasks, the quality of training data has become a bottleneck. While existing systems can generate executable tasks, they often struggle to distinguish between tasks that are too simple (trivial) and those that are effectively unsolvable. This paper asks: how can we use the behavior of solver agents to autonomously construct tasks that are specifically calibrated to be appropriately challenging for learning?
CalibForge introduces an adversarial author-solver loop to synthesize terminal tasks. The system begins by generating a candidate task, which then undergoes structural validation and self-solving. If a candidate passes these initial checks, it enters an adversarial calibration phase using one of two strategies:
If a task fails these criteria, the authoring agent uses the resulting feedback—including interaction trajectories and failure diagnoses—to iteratively revise the task instruction, environment, or verification tests.
Using this process, the authors constructed 5,431 calibrated terminal tasks. When used to train agent models, these tasks significantly outperformed existing datasets. On the Terminal-Bench 2.0 benchmark, models trained with CalibForge data achieved accuracy improvements of up to 24.71 percentage points over base models. Furthermore, the researchers observed strong transferability to out-of-distribution tasks, with significant gains on SWE-bench Pro (for long-horizon software engineering) and Doc2Repo (for repository generation). The results demonstrate that solver-relative learnability is a highly effective metric for curating high-quality, transferable training data.
This work shifts the paradigm of data synthesis from simple executable validation to behavioral calibration. By treating task difficulty as a relative property that depends on the solver's capabilities, CalibForge provides a scalable, automated way to generate training data that pushes the boundaries of agent performance, rather than just providing 'more' data that may be too easy or poorly specified.
Training terminal agents requires executable and verifiable tasks that are not merely solvable, but appropriately challenging for learning. Executable validation establishes feasibility, yet does not reveal how a task behaves relative to a given solver setting. In this paper, we present CalibForge, an autonomous terminal-task synthesis system that uses verified solver behavior to revise candidate tasks through adversarial solver calibration. Multi-solver calibration targets disagreement within a heterogeneous solver pool, whereas contrastive solver calibration targets a designated strong-pass/weak-fail relation; both operationalize a solver-relative learnable zone anchored in demonstrated solvability. Using CalibForge, we construct 5,431 calibrated terminal tasks. Our ablations show that both strategies yield more effective supervision than authoring and validation alone or ordinary single-solver feedback. Models trained on the full collection achieve 32.58% and 47.57% on Terminal-Bench 2.0. The largest improvements over the corresponding base model reach 24.71 percentage points on Terminal-Bench 2.0, 27.68 points on SWE-bench Pro, and 30.04 points on Doc2Repo. Together, these results support solver-relative learnability as a practical target for constructing effective and transferable agent training data.
Alex: So the real contribution here isn't just making more training data—it's building a filter that ensures the data is actually worth learning from.
Sam: Precisely. And the filtering isn't a one-shot process. The system runs what they call an "author-solver loop." An authoring agent creates a task, subjects it to the solver tests, and if the task doesn't land in the learnable zone, the author revises it and tries again. It's an iterative cycle, constantly tuning difficulty until the task is just right.
Alex: Wait—if the selection is that aggressive, doesn't it risk the AI just getting very good at a narrow slice of problems?
Sam: That's a fair concern, and the researchers address it in two ways. First, they use what they call "benchmark decontamination"—they remove any tasks that overlap with the tests used to evaluate the AI, so the model can't accidentally memorise the answers. Second, they use "trajectory distillation." When a high-performing AI successfully solves a task, they record every step of its reasoning—not just the final answer, but the whole path it took to get there. Those reasoning sequences are then used to train smaller models.
Alex: So the student doesn't just see the answer—it sees the thinking behind the answer?
Sam: Exactly. It's the difference between copying someone's homework and watching them work through it step by step. By learning the reasoning process, the student model develops a more general ability to navigate complex problems, which is why the improvements carry over to software engineering tasks the model wasn't specifically trained on.
Alex: This all sounds computationally heavy, though. Running multiple AI agents to test every single task—that must add up.
Sam: It does. That's the main practical limitation. You're not just generating a task; you're running it through a gauntlet of agents to verify it hits the right difficulty level. It takes more time and computing power than simply generating raw data and hoping for the best. The authors are essentially making a deliberate trade-off: spend more resources upfront on curation, and get a more capable model at the end.
Alex: It's like a teacher spending hours refining an exam rather than printing a generic worksheet. More work to prepare, but the students learn more from it.
Sam: That's the logic. And the results support it—models trained on the curated data outperform those trained on larger but unfiltered datasets. It shifts the focus from "more is better" to "better is better."
Alex: Where does this go next? Could an AI eventually manage this curriculum-building process on its own?
Sam: That's the logical direction. Imagine an agent that monitors its own performance and, once it masters a set of tasks, automatically generates a harder set to work on next. It would be a self-improving loop—adapting to its own skill level in real time, without a human needing to design the curriculum at all.
Alex: So instead of us building the school, the AI builds its own school as it goes.
Sam: That's one way to put it. The computational cost is still a real constraint, but the underlying idea—that task construction can be treated as an adaptive, adversarial process—points toward systems that could improve their own capabilities more autonomously over time. It's a meaningful step in that direction.
Alex: It's a useful reminder that how you structure the learning experience can matter just as much as the model itself. Thanks for walking me through this, Sam.
Sam: It was a pleasure. Thanks for listening to ResearchPod.