Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, Jingjing Chen
5 min
Abstract
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
Alex: So it just... stops the practice session when it stops being useful.
Sam: Exactly. The second controller is called a progressive turn-normalized loss controller. Here's the idea: as the model gets better at the early steps, this controller gradually shifts more of the "grade" toward the later, harder turns. It's a measured transition—the model doesn't get thrown into the deep end immediately. It earns its way there as it masters the basics.
Alex: That's a sensible design. But how does the system actually know which parts are "easy" and which are genuinely informative? How does it tell the difference?
Sam: This is where a concept the paper calls "contamination compression" comes in. Imagine reading a textbook where half the sentences are ones you've already memorized. Your brain naturally skims those and focuses on the new material. The AI does something similar. It looks at the history of what it's already done and recognizes familiar patterns—common formatting, repeated phrases, predictable structures. Those familiar parts effectively crowd out the real decision-making, compressing the genuinely important moments into a tiny fraction of the data.
Alex: So the learning signal gets artificially squashed because the model keeps seeing things it already knows?
Sam: Right. And here's the part that makes it worse: failed attempts are often the most predictable of all. When an agent fails, it tends to fall into repetitive, robotic loops—the same wrong action over and over. Because those loops are so predictable, the system treats them as "easy" and learns almost nothing from them. Successful attempts, by contrast, tend to be more varied and complex, so they generate a stronger learning signal.
Alex: So failure is hiding in plain sight. It looks too easy to the model, so the model ignores it.
Sam: Exactly. The system accidentally suppresses the very feedback it needs to correct its mistakes. The paper shows that standard training ends up spending nearly half its total learning budget on just the first few steps of a task—like a teacher who spends the entire semester on the first page of the textbook and never reaches the final exam.
Alex: So TurnOPD is essentially redirecting that budget toward the parts that actually matter. Did it work?
Sam: The results are notable. The approach achieved up to twice the training speed while simultaneously improving accuracy. That's worth paying attention to because it breaks a trade-off that's usually treated as unavoidable—the assumption that if you want better results, you have to spend more time training, and if you want faster training, you accept lower quality.
Alex: And the key insight is that the improvement comes not from doing more, but from stopping the waste.
Sam: That's a clean way to put it. The model isn't working harder—it's working on the right things. By cutting off the tail turns that teach nothing and shifting the grading toward the turns that are genuinely difficult, TurnOPD lets the model spend its effort where it actually moves the needle.
Alex: It's a meaningful reminder that in machine learning, more data and more compute aren't always the answer. Sometimes the bottleneck is just that the training process itself is pointing in the wrong direction.
Sam: And that's a pattern worth watching across the field. A lot of progress in AI right now isn't coming from bigger models or longer training runs—it's coming from researchers asking more carefully: what is the model actually learning from, and is that the right signal?
Alex: Thanks for walking us through it. Thanks for listening to ResearchPod.