When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon

Online imitation learning (IL), such as on-policy distillation, is widely used in LLM post-training to improve model performance. While it often outperforms offline supervised fine-tuning (SFT), the theoretical justification for this advantage remains debated. This paper investigates whether the benefit of online interaction is truly due to mitigating error accumulation over long horizons, or if it stems from the fundamental limitations of learning from a fixed dataset when the student model is misspecified (non-realizable).

The authors analyze imitation learning through a contextual bandit framework, which isolates the effects of misspecification from long-horizon compounding errors. They conduct empirical experiments on math reasoning tasks (Countdown, GSM8K, and DeepScaleR) to compare SFT and on-policy distillation in both realizable and non-realizable settings. Theoretically, they derive information-theoretic lower bounds for offline IL under misspecification and propose a structural condition—the Misspecification–Reward Alignment Condition—to explain when online IL can effectively bridge the performance gap.

1. Realizability Suffices: In settings where the student model can represent the expert (realizable), offline SFT fully recovers expert performance. Online IL offers no significant gains in accuracy or training speed, suggesting that the observed superiority of online methods in practice is not an inherent property of the algorithm but a response to misspecification.
2. The Failure of Discrepancy-Based Analysis: Traditional analyses that bound performance gaps using distributional discrepancies (like total variation or KL divergence) are shown to be overly pessimistic. Large differences between expert and student response distributions do not necessarily translate to large performance gaps, as many stylistic differences are irrelevant to the final reward.
3. The Role of Reward Alignment: The authors identify that online IL succeeds in non-realizable settings when the expert's feedback signal is aligned with the reward function. They prove that under this structural alignment, online IL can achieve near-optimal performance even when the student remains distributionally far from the expert.