Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

On-policy distillation (OPD) is a powerful technique for improving large language models (LLMs) by using a high-capability teacher to provide dense token-level supervision. However, standard OPD typically requires 'teacher consistency'—the condition that the teacher providing the distillation signal must be the same model that generated the demonstrations used for the initial supervised fine-tuning (SFT). This requirement is often impractical, as it limits the ability to independently select the best models for SFT data generation and subsequent distillation. This paper investigates how to decouple these roles without sacrificing performance.

The authors identify that raw disagreement between a teacher and an SFT reference model contains two components: context-specific reasoning evidence and a recurring component related to differences in style, formatting, and reasoning cadence. Because the latter is predictable across rollouts, the authors introduce Lightning OPD 2.0. This method uses cross-fitting to estimate this recurring style bias by constructing lookup tables based on token identity, normalized response position, and reference-policy surprisal. By subtracting this estimated bias from the raw teacher-reference disagreement, the model isolates the reasoning-related signal for the distillation update.

Lightning OPD 2.0 consistently outperforms standard Lightning OPD in cross-teacher settings across both mathematical reasoning and code generation benchmarks. By relaxing the teacher consistency constraint, the authors demonstrate that practitioners can achieve state-of-the-art results even when the SFT data generator and the OPD teacher are selected independently. For instance, starting from a Klear-Reasoner-8B-SFT model, the approach achieved 82.4% on the AIME 2024 benchmark and 63.0% on LiveCodeBench v5.