Emerging Properties in Self-Supervised Vision Transformers

The authors investigate whether self-supervised learning can unlock unique, emerging properties in Vision Transformers (ViT) that are not observed in supervised models or traditional convolutional networks. By treating self-supervised learning as a form of self-distillation without labels—a method they call DINO—they show that ViTs learn to capture explicit object-level information during pretraining.

The DINO framework functions as a student-teacher distillation process. The student network learns to predict the output of a teacher network, where the teacher is a momentum-based ensemble of the student's own previous states. The architecture is notable for its simplicity: it relies on a centering and sharpening mechanism to prevent the common issue of 'model collapse' (where the model outputs the same value regardless of input), while avoiding more complex techniques like batch normalization or contrastive losses. The study specifically evaluates how these features perform on standard tasks like ImageNet linear classification, nearest-neighbor retrieval, and video instance segmentation.

DINO-trained ViTs exhibit striking, class-specific self-attention patterns that naturally delineate objects in an image. Quantitatively, these self-supervised features are highly effective: they achieve 80.1% top-1 accuracy on ImageNet with a ViT-Base architecture using linear evaluation. Perhaps more significantly, these features are excellent k-nearest neighbor (k-NN) classifiers, reaching 78.3% accuracy without needing any finetuning. The researchers also found that reducing the patch size of the input images—without adding parameters—significantly improves the quality of the learned representations.

This paper suggests that self-supervised pretraining is a powerful mechanism for unlocking spatial understanding in Transformers. By demonstrating that ViTs can 'discover' object boundaries and semantic segments without explicit labels, the authors provide a compelling case for using self-supervised ViTs as a foundation for downstream vision tasks that require high-level scene understanding, such as retrieval and segmentation.