When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation

On-Policy Distillation (OPD) is a powerful technique for transferring knowledge from a teacher model to a student by supervising the student's own trajectories. However, existing methods often treat all teacher signals as equally valid if they are confident or salient. This paper investigates a critical failure mode: teacher models often provide strong supervision based on generic language priors, formatting conventions, or stereotyped reasoning patterns rather than the actual task input. These 'spurious signals' create large gradients that do not align with task-improving directions, potentially leading to student bias and suboptimal performance.

The authors propose Spurious-Signal-Aware On-Policy Distillation (SA-OPD). The core innovation is a lightweight proxy for 'input-groundedness.' For every token in a student-generated trajectory, the framework calculates the teacher-student divergence under two conditions: the original input and a 'no-prompt' context. If the divergence remains high even when the input is removed, the signal is identified as being driven by input-agnostic priors. SA-OPD then filters out tokens that simultaneously exhibit low input-groundedness and high absolute distillation divergence, effectively removing high-impact, misleading updates while retaining grounded, task-relevant supervision.

Experimental results across both Large Language Models (LLMs) and Vision-Language Models (VLMs) demonstrate that SA-OPD consistently outperforms standard OPD and existing selective distillation methods. In mathematical reasoning tasks, SA-OPD achieved significant improvements over baselines, including a 3.0-point gain on the Math500 benchmark. Similarly, in visual understanding and reasoning tasks, the framework consistently secured the best performance across all tested benchmarks. The results suggest that filtering for input-groundedness is a critical dimension for stable and effective knowledge distillation.