Enabling agents to learn from experience and internalize it into their policy has become a central problem in self-evolving AI. On-policy self-distillation (OPSD) offers an effective pathway by using a privileged self-teacher to provide dense supervision on the student's own trajectories; however, existing methods still rely heavily on designer-specified privileged artifacts (e.g., answers, feedback, skills, or trajectories), limiting the end-to-end learnability and scalability required for continual self-improvement. In this work, we introduce Latent On-Policy Self-Distillation (LOPD), which, rather than proposing another hand-crafted OPSD variant with a newly prescribed form of privileged context, makes the teacher's privileged context itself learnable end-to-end from experience. Technically, LOPD retrieves relevant experiences and composes them into continuous latent tokens that condition a self-teacher, while the student generates trajectories from the task and interaction history and receives dense token-level supervision at every visited prefix. We further introduce a privileged-margin objective to stabilize and regulate the learning of latent context. Empirically, LOPD demonstrates (I) strong performance, outperforming RLVR and representative OPSD methods including OPSD, SDPO, and Skill-SD across both agentic tool use and code generation; and (II) high learning efficiency, surpassing GRPO and Skill-SD with less than 30% of their rollout budget. Ablation studies further provide direct evidence that making privileged context learnable is necessary for realizing these gains. Together, these results position LOPD as a step toward a more scalable and self-directed paradigm for agent evolution.
Alex: Welcome to another episode of ResearchPod. Today we're looking at how computer scientists are trying to change the way autonomous software agents learn from their own experience.
Sam: So the central question this paper is asking is: why does self-improving AI still depend so heavily on rigid, hand-crafted rules written by human engineers?
Alex: Exactly. Right now, if you want an AI system to learn from its mistakes, a human has to sit down and write out the feedback templates, the hints, the structure of what "a good lesson" even looks like. The system can't figure that out on its own.
Sam: And that becomes a real bottleneck when you're trying to scale up to complex tasks — like writing code or using software tools across many steps.
Alex: Right. So let's back up and think about how these systems are trained in the first place. Imagine you have two versions of the same AI — one acting as a student, one acting as a teacher. The student tries to solve a problem, and the teacher watches and gives feedback on every step.
Sam: That sounds reasonable. So what's the problem?
Alex: The problem is that for the teacher to give useful feedback, it usually needs extra information the student doesn't have — like the verified correct answer, or a pre-written explanation of how to reason through the problem. Those hints are called privileged context. And traditionally, a human engineer writes those hints by hand, using fixed rules.
Sam: So the system is permanently limited by whatever the human designer decided to write down.
Alex: That's the constraint. And the paper's central argument is that if AI is going to genuinely self-improve, it needs to discover those useful lessons on its own — not rely on a human to prescribe them in advance.
Sam: So how does the framework they propose actually do that?
Alex: The approach is called Latent On-Policy Self-Distillation. The key idea is to replace those hand-written hints with something the system learns for itself. Here's how it works. The model keeps a memory bank of its past attempts — old code it wrote, sequences of tool-use steps it tried, that kind of thing.
Sam: But past attempts can be really long and messy. How do you feed all of that into the teacher without overwhelming it?
Alex: Good question — and that's where the compression step comes in. Those long, variable-length records of past attempts get passed through a specialized encoder, which squeezes them down into a small, fixed set of compact summary tokens. Think of it like taking a whole chapter of notes and condensing them into a tight cheat sheet — same essential information, much smaller package.
Sam: And those summary tokens are what the teacher uses instead of human-written hints?
Alex: Exactly. The teacher receives those compact tokens alongside the current problem the student is working on, and treats them like hidden background knowledge — context it can draw on when evaluating the student's steps. No human wrote those tokens. The system learned how to build them.
Sam: So the teacher can now judge the student's work with the benefit of hindsight, automatically.
Alex: Precisely. The student works through a multi-step problem, and then the teacher goes back and re-evaluates those exact steps using the compressed history. The goal is to measure how differently the teacher would have approached each moment compared to the student — and then use that gap to push the student to improve.
Sam: How does the system actually measure that gap?
Alex: There's a mathematical way of measuring how different two probability distributions are — think of it like measuring how far apart two people's guesses are on a multiple-choice test. The system uses a specific version of this measurement, chosen so that the student learns to focus tightly on the behaviors the teacher most strongly supports, rather than spreading its guesses too broadly.
Sam: That keeps the student's responses sharp and confident rather than vague.
Alex: Exactly. And because the teacher is drawing on a compressed summary of real past experience — not a rule a human wrote — the feedback it gives is grounded in what actually happened during training.
Sam: So what does the evidence show about how well this actually works?
Alex: Across ten distinct benchmarks covering tool use and code generation, this approach achieves the strongest aggregate results while using a fraction of the computational trial-and-error budget that standard alternatives require. The latent teacher is extracting a denser, more useful learning signal from each piece of historical experience it visits.
Sam: That efficiency seems significant — it means the system doesn't need to run thousands of extra attempts just to absorb what went wrong.
Alex: That's right. Though it's worth being clear about the trade-offs. The method requires maintaining an external memory bank and continuously retrieving from it during training, which adds computational overhead. And the system's performance is sensitive to how you configure things — how many summary tokens you use, how much history you retrieve. Too few tokens, or poorly chosen settings, and the gains shrink noticeably.
Sam: So it's not a plug-and-play solution. It demands careful setup.
Alex: That balance is important to keep in mind. The broader implication the paper draws is that scalable self-improvement should be built on representations the system learns from experience — not artifacts that human designers author in advance.
Sam: That's a meaningful shift in how you'd think about designing these systems. Instead of asking "what rules should we write," you're asking "how do we let the system discover the rules itself."
Alex: That's the central contribution. Thanks for listening to this look at Latent On-Policy Self-Distillation, here on ResearchPod.