ResearchPod Summary
On-policy distillation (OPD) has emerged as a critical technique in modern LLM post-training, bridging the gap between supervised imitation and reinforcement learning. Unlike standard supervised fine-tuning (SFT) or teacher-forced knowledge distillation, which rely on external or teacher-provided data, OPD trains models on states generated by the student's own policy. This approach ensures the model learns to handle the specific distribution of states it encounters during deployment, but it introduces significant challenges regarding feedback stability and actionability.
Rather than viewing OPD as a single family of loss functions, this paper proposes a taxonomy based on the feedback-to-update path. The core insight is that OPD methods can be decomposed into specific design variables: how states are sampled, the support used for comparison, the divergence chosen, and how the resulting signal is routed into a parameter update. By analyzing the underlying formulas of direct distributional losses and policy-gradient-style updates, the author identifies two critical, often conflated mechanisms:
This framework allows researchers to move beyond surface-level labels (e.g., "sampled-token" or "top-k") and instead evaluate OPD methods based on their specific intervention points. By separating these variables, the paper provides a diagnostic toolkit for identifying why certain OPD implementations fail—such as issues with state compatibility, support truncation, or high-variance updates—and offers a structured checklist for reporting and designing more robust post-training pipelines.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.