Guibin Zhang, Jiayang Lyu, Ran Sun, Xinlei Yu, Haoyu Zhao, Qibing Ren, Shuicheng Yan
5 min
Abstract
Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.
Sam: And those summary tokens are what the teacher uses instead of human-written hints?
Alex: Exactly. The teacher receives those compact tokens alongside the current problem the student is working on, and treats them like hidden background knowledge — context it can draw on when evaluating the student's steps. No human wrote those tokens. The system learned how to build them.
Sam: So the teacher can now judge the student's work with the benefit of hindsight, automatically.
Alex: Precisely. The student works through a multi-step problem, and then the teacher goes back and re-evaluates those exact steps using the compressed history. The goal is to measure how differently the teacher would have approached each moment compared to the student — and then use that gap to push the student to improve.
Sam: How does the system actually measure that gap?
Alex: There's a mathematical way of measuring how different two probability distributions are — think of it like measuring how far apart two people's guesses are on a multiple-choice test. The system uses a specific version of this measurement, chosen so that the student learns to focus tightly on the behaviors the teacher most strongly supports, rather than spreading its guesses too broadly.
Sam: That keeps the student's responses sharp and confident rather than vague.
Alex: Exactly. And because the teacher is drawing on a compressed summary of real past experience — not a rule a human wrote — the feedback it gives is grounded in what actually happened during training.
Sam: So what does the evidence show about how well this actually works?
Alex: Across ten distinct benchmarks covering tool use and code generation, this approach achieves the strongest aggregate results while using a fraction of the computational trial-and-error budget that standard alternatives require. The latent teacher is extracting a denser, more useful learning signal from each piece of historical experience it visits.
Sam: That efficiency seems significant — it means the system doesn't need to run thousands of extra attempts just to absorb what went wrong.
Alex: That's right. Though it's worth being clear about the trade-offs. The method requires maintaining an external memory bank and continuously retrieving from it during training, which adds computational overhead. And the system's performance is sensitive to how you configure things — how many summary tokens you use, how much history you retrieve. Too few tokens, or poorly chosen settings, and the gains shrink noticeably.
Sam: So it's not a plug-and-play solution. It demands careful setup.
Alex: That balance is important to keep in mind. The broader implication the paper draws is that scalable self-improvement should be built on representations the system learns from experience — not artifacts that human designers author in advance.
Sam: That's a meaningful shift in how you'd think about designing these systems. Instead of asking "what rules should we write," you're asking "how do we let the system discover the rules itself."
Alex: That's the central contribution. Thanks for listening to this look at Latent On-Policy Self-Distillation, here on ResearchPod.