Jianyu Wu, Yizhou Wang, Encheng Su, Chen Tang, Shixiang Tang
4 min
On-policy self-distillation (OPSD) is a popular post-training technique where a student model learns from a teacher model that has access to privileged information, such as ground-truth references or tool outputs. However, this creates an information asymmetry: the teacher uses privileged data that the student cannot access at inference time. This leads to a phenomenon called privilege illusion, where the student attempts to mimic the teacher's behavior by making unsupported claims or continuing derivations as if the privileged information were present, ultimately degrading performance.
The authors identify this information asymmetry as the root cause of the privilege illusion. To address it, they introduce Dual-Anchored Policy Distillation (DAPD), which ensures that the student and teacher distributions are aligned under matched information availability. DAPD consists of two main components:
Experiments across six benchmarks (covering reasoning, coding, and instruction following) show that DAPD significantly outperforms standard OPSD. On the Qwen3-4B model, DAPD improves performance by an average of +2.00 points. Crucially, while the benefits of standard OPSD often diminish as model scale increases, DAPD maintains consistent improvements across scales, achieving gains of +2.69 at 4B and +2.78 at 32B. By directly addressing the structural mismatch between teacher and student, DAPD provides a more robust framework for distilling reasoning capabilities into language models.
On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.
Sam: Wait—if the training process is this selective about what the model learns from, doesn't it risk becoming too narrowly tuned? Like, overfitting to a very specific slice of the data?
Alex: That's a legitimate concern, and it's one the paper addresses directly. To prevent that, they use what they call "Dual-Source Anchoring." Rather than relying on a single learning signal, the model balances two sources: the reliable teacher reference on one side, and the model's own attempts on the other.
Sam: So it's not just following the teacher—it's also learning from its own mistakes?
Alex: Correct. By holding both in balance, the model learns to be accurate while staying within the bounds of what it can actually do on its own. It can't just memorize the teacher's style, because it's also being held accountable to its own reasoning.
Sam: It sounds like they've built a system that forces the model to genuinely understand the reasoning, rather than guess based on hidden clues.
Alex: That is the core mechanism. And the paper goes further than just demonstrating it empirically—they provide a formal mathematical proof showing that this "bridge" between the student's self-generated reasoning and the teacher's reference logic keeps the error bounded. So it's not just a heuristic that happens to work; there's a theoretical basis for why it should hold.
Sam: That's a meaningful distinction. Though I imagine all of this costs more during training?
Alex: That is the primary trade-off. Because the model has to calculate multiple learning paths simultaneously, training is more computationally expensive. But once training is complete, using the model is no slower than any standard approach.
Sam: So you pay the cost once, during training, and get a more reliable model on the other side.
Alex: That's the paper's argument, yes. By addressing the root cause—the gap between training conditions and real-world conditions—DAPD offers a more principled approach to building models that reason from their own understanding, rather than from hidden guidance that won't be there when it matters. Thanks for listening to ResearchPod.