ResearchPod Summary
On-policy self-distillation (OPSD) is a popular post-training technique where a student model learns from a teacher model that has access to privileged information, such as ground-truth references or tool outputs. However, this creates an information asymmetry: the teacher uses privileged data that the student cannot access at inference time. This leads to a phenomenon called privilege illusion, where the student attempts to mimic the teacher's behavior by making unsupported claims or continuing derivations as if the privileged information were present, ultimately degrading performance.
The authors identify this information asymmetry as the root cause of the privilege illusion. To address it, they introduce Dual-Anchored Policy Distillation (DAPD), which ensures that the student and teacher distributions are aligned under matched information availability. DAPD consists of two main components:
Experiments across six benchmarks (covering reasoning, coding, and instruction following) show that DAPD significantly outperforms standard OPSD. On the Qwen3-4B model, DAPD improves performance by an average of +2.00 points. Crucially, while the benefits of standard OPSD often diminish as model scale increases, DAPD maintains consistent improvements across scales, achieving gains of +2.69 at 4B and +2.78 at 32B. By directly addressing the structural mismatch between teacher and student, DAPD provides a more robust framework for distilling reasoning capabilities into language models.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper called "DAPD: Dual-Anchored Policy Distillation," and it tackles a persistent problem in how we train modern language models—why they sometimes make things up, even after being trained on correct answers.
Sam: So the question is: why do these models hallucinate, even when they've seen the right answers during training?
Alex: Exactly. Here's the core issue. When we train these models, we often hand them what you might call an "answer key"—extra information that helps them learn faster. But when the model is actually deployed and someone asks it a question, that answer key is gone.
Sam: And because the model got used to having that key during training, it tries to act as if it still has it—even when it doesn't?
Alex: Precisely. The researchers call this the "privilege illusion." The model has learned to lean on a crutch that simply isn't there during a real test. And when it reaches for that crutch and finds nothing, it fills the gap with confident-sounding but unsupported claims. Think of a student who only ever solved math problems by checking the back of the textbook. Take the book away, and they're lost—but they might still write down an answer anyway.
Sam: So how does this paper suggest we fix that?
Alex: They propose a framework called Dual-Anchored Policy Distillation, or DAPD. The central idea is to close what they call the "information asymmetry"—the gap between what the teacher model knows during training and what the student model knows when it's actually being used.
Sam: So they're trying to make the training environment match the real-world environment more closely?
Alex: That's right. The way they do it is by having the model generate its own internal reference—what the paper calls a "Self" distribution. Instead of just copying the teacher's final answer, the student has to align its own reasoning process with the teacher's, but under the same information constraints. It's a bit like a tutor who doesn't just show you the solution, but forces you to work through the problem yourself and then checks that your thinking matches theirs step by step.
Sam: So the student can't just pattern-match to the answer key. It has to actually develop the reasoning.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: Exactly. They use two learning paths simultaneously: one for when the model has no extra information at all, and one for when it does. Keeping both paths active prevents the student from developing habits that only work when the answer key is present.
Sam: Does this actually reduce hallucinations in practice?
Alex: The paper suggests it does. By keeping the information available to the student and the teacher consistent, they report a meaningful reduction in unsupported claims compared to standard training methods. And importantly, the gains hold as the models get larger—up to around 32 billion parameters—whereas other approaches tend to lose their effectiveness at that scale.
Sam: Wait—if the training process is this selective about what the model learns from, doesn't it risk becoming too narrowly tuned? Like, overfitting to a very specific slice of the data?
Alex: That's a legitimate concern, and it's one the paper addresses directly. To prevent that, they use what they call "Dual-Source Anchoring." Rather than relying on a single learning signal, the model balances two sources: the reliable teacher reference on one side, and the model's own attempts on the other.
Sam: So it's not just following the teacher—it's also learning from its own mistakes?
Alex: Correct. By holding both in balance, the model learns to be accurate while staying within the bounds of what it can actually do on its own. It can't just memorize the teacher's style, because it's also being held accountable to its own reasoning.
Sam: It sounds like they've built a system that forces the model to genuinely understand the reasoning, rather than guess based on hidden clues.
Alex: That is the core mechanism. And the paper goes further than just demonstrating it empirically—they provide a formal mathematical proof showing that this "bridge" between the student's self-generated reasoning and the teacher's reference logic keeps the error bounded. So it's not just a heuristic that happens to work; there's a theoretical basis for why it should hold.
Sam: That's a meaningful distinction. Though I imagine all of this costs more during training?
Alex: That is the primary trade-off. Because the model has to calculate multiple learning paths simultaneously, training is more computationally expensive. But once training is complete, using the model is no slower than any standard approach.
Sam: So you pay the cost once, during training, and get a more reliable model on the other side.
Alex: That's the paper's argument, yes. By addressing the root cause—the gap between training conditions and real-world conditions—DAPD offers a more principled approach to building models that reason from their own understanding, rather than from hidden guidance that won't be there when it matters. Thanks for listening to ResearchPod.