Yuhang Zhou, Kai Zheng, Haoling Li, Dengyun Peng, Can Xu, Jingjing Chen
5 min
Language models are increasingly deployed as agents for multi-turn planning, tool use, and environment interaction. On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, providing dense feedback without sparse reward signals. However, applying OPD to long-horizon agent tasks is challenging. The authors identify two main inefficiencies in vanilla agent OPD: first, full-horizon rollouts often waste computational resources on tail turns that provide weak and noisy KL supervision; second, trajectory-level KL objectives concentrate most of the optimization loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned.
The authors systematically analyze supervision signals along the interaction trajectory and find that raw reverse-KL is heavily skewed toward early turns. Furthermore, outcome-separation analyses reveal that deep-turn KL lacks outcome-predictive power and can even invert, where successful trajectories show higher per-turn KL than failed ones. To explain this, the paper formalizes a contamination-compression mechanism. As student-generated contexts grow longer, a larger portion of the next-token distribution becomes self-determined or forced by formatting constraints and repetition. This compressed mass artificially suppresses measured KL even when underlying policy disagreements remain, explaining why late-turn signals decay and why failed trajectories can exhibit heavily suppressed KL due to degenerate loops.
To overcome these external mismatches (wasted tail compute) and internal mismatches (shallow token dominance), the authors propose TurnOPD. This framework consists of two main budget controllers:
Experiments on multi-turn agent benchmarks—including ALFWorld, WebShop, and Multi-Hop Search—demonstrate that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets. By advancing the accuracy-time frontier, TurnOPD provides comparable or better performance with up to a 2.29x speedup in training time compared to vanilla OPD baselines.
On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
Alex: So it just... stops the practice session when it stops being useful.
Sam: Exactly. The second controller is called a progressive turn-normalized loss controller. Here's the idea: as the model gets better at the early steps, this controller gradually shifts more of the "grade" toward the later, harder turns. It's a measured transition—the model doesn't get thrown into the deep end immediately. It earns its way there as it masters the basics.
Alex: That's a sensible design. But how does the system actually know which parts are "easy" and which are genuinely informative? How does it tell the difference?
Sam: This is where a concept the paper calls "contamination compression" comes in. Imagine reading a textbook where half the sentences are ones you've already memorized. Your brain naturally skims those and focuses on the new material. The AI does something similar. It looks at the history of what it's already done and recognizes familiar patterns—common formatting, repeated phrases, predictable structures. Those familiar parts effectively crowd out the real decision-making, compressing the genuinely important moments into a tiny fraction of the data.
Alex: So the learning signal gets artificially squashed because the model keeps seeing things it already knows?
Sam: Right. And here's the part that makes it worse: failed attempts are often the most predictable of all. When an agent fails, it tends to fall into repetitive, robotic loops—the same wrong action over and over. Because those loops are so predictable, the system treats them as "easy" and learns almost nothing from them. Successful attempts, by contrast, tend to be more varied and complex, so they generate a stronger learning signal.
Alex: So failure is hiding in plain sight. It looks too easy to the model, so the model ignores it.
Sam: Exactly. The system accidentally suppresses the very feedback it needs to correct its mistakes. The paper shows that standard training ends up spending nearly half its total learning budget on just the first few steps of a task—like a teacher who spends the entire semester on the first page of the textbook and never reaches the final exam.
Alex: So TurnOPD is essentially redirecting that budget toward the parts that actually matter. Did it work?
Sam: The results are notable. The approach achieved up to twice the training speed while simultaneously improving accuracy. That's worth paying attention to because it breaks a trade-off that's usually treated as unavoidable—the assumption that if you want better results, you have to spend more time training, and if you want faster training, you accept lower quality.
Alex: And the key insight is that the improvement comes not from doing more, but from stopping the waste.
Sam: That's a clean way to put it. The model isn't working harder—it's working on the right things. By cutting off the tail turns that teach nothing and shifting the grading toward the turns that are genuinely difficult, TurnOPD lets the model spend its effort where it actually moves the needle.
Alex: It's a meaningful reminder that in machine learning, more data and more compute aren't always the answer. Sometimes the bottleneck is just that the training process itself is pointing in the wrong direction.
Sam: And that's a pattern worth watching across the field. A lot of progress in AI right now isn't coming from bigger models or longer training runs—it's coming from researchers asking more carefully: what is the model actually learning from, and is that the right signal?
Alex: Thanks for walking us through it. Thanks for listening to ResearchPod.