Scaling Linear Mode Connectivity and Merging to Billion Parameter Pretrained Transformers

As deep learning models grow in size, understanding the geometric relationship between independently trained checkpoints becomes critical for efficient model reuse and merging. This paper investigates whether Linear Mode Connectivity (LMC)—the property where a linear path between two models maintains low loss—can be scaled to billion-parameter Transformers by resolving the complex, function-preserving symmetries inherent in their architecture.

The authors propose a framework called Dual Learned Matching (LMC-DM). Unlike previous methods that align one model to a fixed target, this approach treats both models as flexible entities. The researchers define a comprehensive family of functionality-preserving transformations, including normalization absorption, residual-space rotations, attention-head permutations, and feed-forward network (FFN) scaling. By parameterizing these symmetries (using techniques like the Cayley transform for orthogonal matrices), they enable a bidirectional optimization process where both models learn to align themselves toward a shared, low-loss linear interpolation path.

The dual learning procedure consistently outperforms existing one-sided matching techniques. For language models, the authors demonstrate near-zero loss barriers on the WikiText dataset for models up to 160M parameters, a significant improvement over prior state-of-the-art. In the vision domain, a ViT-L model maintains over 69% ImageNet top-1 accuracy throughout the entire interpolation path. Even for modern billion-parameter Large Language Models (LLMs), the method successfully reduces the loss barrier, suggesting that large-scale Transformers possess a latent, connectable structure that can be unlocked through proper symmetry alignment.