AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE

Can a standard Transformer architecture, which has revolutionized natural language processing, be applied directly to images for high-performance computer vision tasks without relying on the architectural components of convolutional neural networks (CNNs)?

The researchers introduce the Vision Transformer (ViT), which treats an image as a sequence of patches, similar to how a Transformer treats a sentence as a sequence of words. The image is divided into fixed-size patches, which are then flattened and linearly embedded. These embeddings, along with a learnable classification token and positional information, are fed into a standard Transformer encoder. The model is pre-trained on large-scale datasets (such as JFT-300M or ImageNet-21k) and then fine-tuned on smaller, downstream image recognition benchmarks.

ViT demonstrates that large-scale training can effectively compensate for the lack of image-specific inductive biases. When pre-trained on sufficient data, ViT matches or exceeds the performance of state-of-the-art CNNs across various benchmarks, including ImageNet and the VTAB suite, while requiring significantly fewer computational resources to train. The study also highlights that while ViT underperforms on smaller datasets, its performance scales impressively as both the dataset size and model capacity increase.

This work challenges the long-standing dominance of CNNs in computer vision. By demonstrating that a pure Transformer can achieve superior results, it simplifies the architecture of vision models and suggests that scaling laws observed in language models also apply to visual recognition. This opens the door for unified architectures that can handle both text and images with minimal modification.