Single-Teacher View Augmentation: Enhancing Knowledge Distillation with Student-Guided Perturbations

Knowledge distillation (KD) typically relies on a single, fixed teacher model, which limits the diversity of supervisory signals provided to the student. While multi-teacher distillation can improve performance by aggregating multiple perspectives, it is computationally expensive. Existing single-teacher augmentation methods either use random noise, which lacks semantic grounding, or require complex, multi-stage training processes to generate structured views. This paper asks: can we generate diverse, semantically meaningful teacher views using a single-stage, parameter-efficient framework?

The authors propose Shift-Augmented Knowledge Distillation (SAKD). Instead of using static teacher features to generate perturbations, SAKD uses the student's own evolving intermediate features as a dynamic condition. This allows the generator to adapt as the student learns. To create multiple distinct views without the cost of multiple generators, the authors introduce a parameter-free cyclic shift operation. This operation permutes the base perturbation vector to produce diverse views while maintaining constant parameter complexity. The entire framework is trained end-to-end in a single stage, using a combination of standard KD loss, a consistency loss to ground perturbations in teacher knowledge, and a diversity loss to ensure the views provide complementary information.

SAKD consistently outperforms existing single-stage random perturbation methods and achieves accuracy comparable to more complex, two-stage reconstruction-based approaches. By conditioning on student features, the framework creates a co-evolution loop where the student actively participates in its own supervision. Experiments on CIFAR-100 and ImageNet demonstrate that SAKD is highly efficient, requiring significantly fewer parameters and no pre-training, making it a practical solution for resource-constrained environments.