VISUAL CONTRASTIVE SELF-DISTILLATION

On-policy self-distillation (OPSD) is a powerful technique for training vision-language models (VLMs) by having a model learn from its own outputs. However, effective OPSD typically requires an asymmetric information gap—where the teacher has access to privileged data (like answers or cropped evidence) that the student does not—to ensure the teacher provides a stronger learning signal. This paper asks: can we construct this necessary asymmetry using only input conditioning, without requiring any external labels, reasoning traces, or auxiliary visual evidence?

The authors propose Visual Contrastive Self-Distillation (VCSD). At each step of the student's generation, the model's exponential moving average (EMA) teacher evaluates the next-token distribution under two conditions: the original image and a content-erased (black) control image. By calculating the log-probability difference between these two distributions, the model identifies which tokens are specifically supported by the visual content. This contrast is then used to sharpen the teacher's original-image distribution, creating a refined target that is distilled into the student via forward KL divergence.

VCSD consistently outperforms standard OPSD baselines across multiple model scales (2B to 9B) and architectures (Qwen3-VL and Qwen3.5). On Qwen3-VL, the method significantly improved aggregate performance across seven benchmarks, for instance, raising the 8B model's score from 72.51% to 76.26%. Because VCSD derives its training signal entirely from the input image and its erased counterpart, it avoids the overhead and complexity of external teachers, privileged answers, or additional inference-time costs.

This work demonstrates that high-quality supervision can be generated internally by exploiting the model's own sensitivity to visual input. By replacing expensive or unavailable privileged information with a simple content-erased contrast, VCSD provides a scalable, efficient way to improve vision-language models during post-training, making it easier to enhance performance without needing curated datasets or complex multi-stage training pipelines.