Understanding Knowledge Distillation in Post-Training: When It Helps and When It Fails

This paper investigates the effectiveness of Knowledge Distillation (KD) during the post-training phase of Large Language Models (LLMs). While KD is a well-established technique for model compression, its utility in the context of general instruction-following—where models are trained on large-scale datasets—has remained under-explored. The authors conduct a systematic study to determine when KD provides a meaningful performance boost over standard Supervised Fine-Tuning (SFT) and how data scale and teacher quality influence these outcomes.

The researchers find that KD is most beneficial in low-data regimes, where it helps the student model learn more efficiently than SFT alone. However, as the size of the training dataset grows, the performance gap between KD and SFT narrows, eventually becoming negligible. This suggests that in large-data settings, the student model is already capable of recovering the teacher's knowledge directly from the training labels.

Crucially, the authors demonstrate that this limitation can be overcome by using a stronger, more capable teacher model (e.g., an instruction-tuned 70B model). When the teacher possesses knowledge that exceeds the information contained in the training data, KD continues to provide substantial performance gains even at large data scales.

To address real-world scenarios where high-quality labeled data is scarce, the authors propose a two-stage KD strategy. This approach first uses synthetic, teacher-labeled data to expose the student to a broader range of instruction styles, followed by a refinement stage using a smaller set of high-quality human annotations. This method consistently improves performance across domain-specific tasks like translation, summarization, and scientific reasoning, offering a practical blueprint for building efficient, compact models in data-constrained environments.