Emergent Capabilities Arise Randomly from Learning Sparse Attention Patterns

This paper investigates the mechanistic origins of emergent capabilities in transformer language models. While neural scaling laws suggest smooth improvements in loss, many downstream capabilities (like in-context learning) appear abruptly at specific scales. The authors seek to understand what internal changes occur within the model during these sudden transitions and why they appear to be stochastic across different initialization seeds.

To isolate the mechanics of emergence, the authors analyze the Pythia suite of language models on tasks like repetition and indirect object identification. They employ activation patching—a technique where specific attention heads from a post-emergence model are inserted into a pre-emergence model—to determine if learning specific attention patterns is the primary driver of capability acquisition. Additionally, they train transformers on synthetic datasets (linear maps and cellular automata) where the ground-truth attention patterns are known, allowing for precise measurement of how context length, sparsity, and architectural choices influence the learning of these patterns.

• Stochastic Emergence: Capabilities emerge at different points during training depending on the initialization seed, with larger models generally acquiring them earlier and more consistently.
• Attention as a Bottleneck: The abrupt jump in performance coincides with the sudden acquisition of task-relevant attention patterns. Patching these patterns into earlier checkpoints can recover performance, proving that the inability to learn these patterns is the primary bottleneck.
• Synthetic Task Dynamics: In controlled settings, the difficulty of learning increases with the sparsity of the required attention pattern. Scaling the number of attention heads improves efficiency, while increasing head dimension yields diminishing returns.
• Architectural Alternatives: On specific synthetic tasks, the MLP-Mixer architecture outperforms standard transformers, suggesting that alternative token-mixing mechanisms may be more sample-efficient for complex attention patterns.