PASS THE BATON: TRAJECTORY-RELAYED ON-POLICY DISTILLATION

On-policy distillation (OPD) is a popular technique for training smaller student models by having them generate trajectories and receiving token-level supervision from a stronger teacher. However, OPD suffers from prefix failure: if a student makes an early mistake, it continues to generate incorrect reasoning, leading to unreliable supervision and wasted compute. This paper asks: can we detect these failure points online and intervene to steer the student back to a correct reasoning path without requiring external labels or reward models?

The authors identify a consistent asymmetry between teacher and student behavior: when a student's reasoning goes off-track, the teacher often attempts to redirect using specific reflection tokens (e.g., "Wait," "But," "However"), whereas the student continues along its original, incorrect path.

Based on this, the authors propose Relay-OPD. During training, the system monitors the student's generation for a handoff trigger—a state where the teacher would use a reflection token but the student would not. When this trigger fires, the teacher briefly takes over to generate a "teacher leg" that corrects the reasoning. The student then resumes generation. To keep the training efficient and close to the student's own policy, the authors implement a relay budget that limits the number and length of these interventions, focusing them on early, critical positions in the reasoning chain.

Relay-OPD significantly outperforms standard OPD and existing baselines like FastOPD across eight mathematical reasoning benchmarks. For a 1.7B parameter student, it improves accuracy by +5.73% over standard OPD and +1.49% over FastOPD. Furthermore, by terminating trajectories early when they are no longer productive and focusing on high-quality interventions, the method reduces the average training trajectory length by over 50%.