REViT: Roto-reflection Equivariant Convolutional Vision Transformer

Traditional Vision Transformers (ViTs) often struggle with rotational and reflectional symmetries because they rely on positional encodings that are not inherently equivariant to these transformations. The authors investigate how to design a Vision Transformer that maintains roto-reflection equivariance—ensuring that if an input image is rotated or flipped, the internal feature representations transform in a predictable, corresponding manner—without the complexity of traditional relative position encoding.

To achieve this, the authors propose REViT, which replaces standard linear patch projections with convolutional projections. The architecture incorporates two key components:

1. Lifting Layer: This layer maps input images from 2D Euclidean space into a higher-dimensional feature space that explicitly accounts for both spatial position and discrete orientations (e.g., 90-degree rotations).
2. Group Convolutional Self-Attention (G-CSA): Instead of standard self-attention, the model uses G-CSA, which performs attention operations over both spatial and group dimensions. By using convolutional projections for queries, keys, and values, the model avoids the need for explicit positional encodings, as the convolutional structure naturally preserves spatial and group-wise relationships.

The study demonstrates that REViT outperforms existing discrete roto-reflection equivariant models on standard image classification benchmarks. By removing the need for relative position encoding, the proposed architecture achieves higher accuracy with fewer parameters compared to previous group-equivariant ViT designs. Furthermore, the authors show that the model is scalable, successfully applying it to the large-scale ImageNet-1k dataset.

This work provides a more efficient and simpler path for incorporating geometric symmetries into transformer-based architectures. By leveraging the strengths of convolutional neural networks (shift equivariance) within the transformer framework, REViT offers a robust backbone for computer vision tasks where object orientation is critical, such as medical imaging, robotics, and remote sensing.