On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at how we train AI agents—those digital assistants designed to navigate complex tasks like browsing the web or filling out forms online.
Sam: We're discussing a paper on something called On-Policy Distillation, or OPD. The central puzzle is this: AI agents often get worse the longer a task runs. Not because they forget things, but because we accidentally train them to be great at the beginning while ignoring the critical decisions that come later.
Alex: So we're wasting training time on the wrong parts of the task?
Sam: Exactly. Think of a student learning to solve a multi-step math problem. If they only ever practice the first step, they'll get very good at setting up the problem—but they'll fall apart when it comes to the final calculation. Standard training does something similar. It focuses too much on the early, easy steps and leaves the deep, informative decisions at the end almost untouched.
Alex: So the system gets "distracted" by the beginning and never really learns how to finish?
Sam: That's the core of it. And there are two specific inefficiencies the paper identifies. The first is what happens at the end of a training run. When you train an AI agent, you let it work through a complete task from start to finish—that's called a "rollout." The problem is, agents keep going even after they've clearly succeeded or failed. They're burning computing power on what the paper calls "tail turns"—the last few steps that teach the model nothing new.
Alex: Like running a race for an extra lap after you've already crossed the finish line.
Sam: Precisely. The second inefficiency is subtler. The training signal—the feedback that tells the model it made a mistake—gets spread evenly across every single word the agent produces. Because the easy, predictable parts of a task happen most often, they soak up the majority of that signal. The complex, genuinely difficult decisions at the end are left with almost nothing to learn from.
Alex: So the model is being graded mostly on the easy stuff, and the hard stuff barely counts?
Sam: That's the mismatch. And to fix both problems, the researchers developed a system called TurnOPD. Think of it like a smart budget manager for the training process. It uses two separate controllers to make sure the learning effort goes where it actually matters.
Alex: How does each controller work?
Sam: The first is an adaptive rollout-depth controller. It monitors how the agent is performing and stops a rollout early once the agent has clearly learned enough from the early steps—cutting off those wasteful tail turns before they drain resources.
Alex: So it just... stops the practice session when it stops being useful.
Sam: Exactly. The second controller is called a progressive turn-normalized loss controller. Here's the idea: as the model gets better at the early steps, this controller gradually shifts more of the "grade" toward the later, harder turns. It's a measured transition—the model doesn't get thrown into the deep end immediately. It earns its way there as it masters the basics.
Alex: That's a sensible design. But how does the system actually know which parts are "easy" and which are genuinely informative? How does it tell the difference?
Sam: This is where a concept the paper calls "contamination compression" comes in. Imagine reading a textbook where half the sentences are ones you've already memorized. Your brain naturally skims those and focuses on the new material. The AI does something similar. It looks at the history of what it's already done and recognizes familiar patterns—common formatting, repeated phrases, predictable structures. Those familiar parts effectively crowd out the real decision-making, compressing the genuinely important moments into a tiny fraction of the data.
Alex: So the learning signal gets artificially squashed because the model keeps seeing things it already knows?
Sam: Right. And here's the part that makes it worse: failed attempts are often the most predictable of all. When an agent fails, it tends to fall into repetitive, robotic loops—the same wrong action over and over. Because those loops are so predictable, the system treats them as "easy" and learns almost nothing from them. Successful attempts, by contrast, tend to be more varied and complex, so they generate a stronger learning signal.
Alex: So failure is hiding in plain sight. It looks too easy to the model, so the model ignores it.
Sam: Exactly. The system accidentally suppresses the very feedback it needs to correct its mistakes. The paper shows that standard training ends up spending nearly half its total learning budget on just the first few steps of a task—like a teacher who spends the entire semester on the first page of the textbook and never reaches the final exam.
Alex: So TurnOPD is essentially redirecting that budget toward the parts that actually matter. Did it work?
Sam: The results are notable. The approach achieved up to twice the training speed while simultaneously improving accuracy. That's worth paying attention to because it breaks a trade-off that's usually treated as unavoidable—the assumption that if you want better results, you have to spend more time training, and if you want faster training, you accept lower quality.
Alex: And the key insight is that the improvement comes not from doing more, but from stopping the waste.
Sam: That's a clean way to put it. The model isn't working harder—it's working on the right things. By cutting off the tail turns that teach nothing and shifting the grading toward the turns that are genuinely difficult, TurnOPD lets the model spend its effort where it actually moves the needle.
Alex: It's a meaningful reminder that in machine learning, more data and more compute aren't always the answer. Sometimes the bottleneck is just that the training process itself is pointing in the wrong direction.
Sam: And that's a pattern worth watching across the field. A lot of progress in AI right now isn't coming from bigger models or longer training runs—it's coming from researchers asking more carefully: what is the model actually learning from, and is that the right signal?
Alex: Thanks for walking us through it. Thanks for listening to ResearchPod.