AsyncOPD: How Stale Can On-Policy Distillation Be?

Research Question and Approach

On-policy distillation (OPD) is a critical technique for training language models, but it suffers from a systems bottleneck where the learner must wait for fresh rollouts. While asynchronous training can decouple rollout generation from learner updates to improve throughput, it introduces stale data. This paper provides the first systematic study of how staleness affects OPD, specifically in practical settings where full teacher logits are too expensive to store, necessitating the use of finite teacher-score caches.

The researchers evaluate how different KL divergences (forward vs. reverse) behave under staleness. They compare standard asynchronous reinforcement learning (RL) stabilization methods—such as PPO-style clipping and decoupled updates—against a simpler, OPD-specific surrogate that recomputes the reverse-KL advantage at learner time. Finally, they introduce AsyncOPD, an asynchronous pipeline that optimizes the bias-variance tradeoff using multi-sample Monte Carlo estimation.

The study reveals that the direction of the KL divergence significantly impacts robustness to stale data. Forward-KL OPD is teacher-weighted and relatively robust to stale rollouts. In contrast, reverse-KL OPD is student-weighted and highly vulnerable to staleness, as the current student's action distribution may shift outside the cached teacher-scored support.

For the vulnerable reverse-KL case, the authors find that advanced asynchronous RL methods do not outperform a straightforward approach: recomputing the reverse-KL advantage under the current student policy at the time of the learner update, without clipping. Furthermore, they demonstrate that while one-sample Monte Carlo estimation is correctable, it suffers from high variance; their proposed multi-sample Monte Carlo approach effectively reduces this variance while preserving correctness. AsyncOPD achieves a 1.6x to 3.8x improvement in training throughput compared to synchronous training while maintaining comparable accuracy.