SR-OPSD: Self-Referenced On-Policy Self-Distillation

On-policy self-distillation (OPSD) is a powerful technique for training large language models (LLMs) by using the model's own outputs—conditioned on privileged information or feedback—as a teacher. However, because the student and the self-teacher co-evolve, the training objective often suffers from instability and distributional collapse. This paper investigates how to stabilize this process by reformulating the distillation objective to separate the target's definition from the projection method.

The authors introduce Self-Referenced On-Policy Self-Distillation (SR-OPSD). The framework makes two primary modifications to standard OPSD:

1. Reference-Anchored Targets: Instead of relying solely on a moving self-teacher, the target is constructed as a geometric interpolation between the self-teacher and a fixed reference policy. This provides a stable anchor that prevents the target from drifting too far during training.
2. Rényi Projection: The authors replace standard KL-divergence objectives with Rényi divergence. This allows the user to control the projection geometry—specifically, how sensitive the student's updates are to token-level density ratios—independently of the target's location.

By using a variational characterization, the authors show that these design choices allow for a more robust optimization objective that remains stable even over long training durations.

SR-OPSD consistently outperforms existing OPSD and reinforcement learning baselines (such as GRPO and SDPO) across scientific reasoning, mathematical problem-solving, and code generation tasks. Empirical results demonstrate that while standard OPSD methods often peak early and then degrade due to instability, SR-OPSD maintains stable policy entropy and performance throughout the training process. The method is shown to be effective across multiple model architectures, including Qwen3 and OLMo.