On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

Recent research has sought to explain the internal mechanisms of transformer models by viewing the inference process as a continuous-time dynamical system. While many studies have modeled these systems as Wasserstein gradient flows, such approaches often require restrictive assumptions on the weight matrices (e.g., symmetry or specific alignment between query, key, and value matrices). This paper introduces a new analytical framework by mapping the inference dynamics of linear self-attention to interacting particle systems, specifically Kuramoto-type models.

The authors show that in an embedding dimension of two, the linear self-attention mechanism can be reformulated as a system of particles on a circle interacting through a second-harmonic coupling. By applying Watanabe–Strogatz theory, they demonstrate that these dynamics are intrinsically low-dimensional. This reduction allows the researchers to analyze the system's long-term behavior using the Ott–Antonsen (OA) manifold, a special subspace where the complex evolution of the entire system is captured by a single, tractable ordinary differential equation.

By analyzing this reduced system, the authors identify four distinct regimes of behavior depending on the configuration of the weight matrices. Beyond the commonly observed phenomenon of token clustering (where representations collapse toward a single point), the model reveals that specific parameter choices can induce sustained oscillations and bifurcations. The study further establishes a structural stability result, proving that these behaviors are not merely artifacts of the OA manifold but are robust to perturbations in initial conditions. Numerical experiments suggest that these qualitative dynamics—clustering, oscillation, and bifurcation—persist in higher-dimensional embedding spaces, providing a broader understanding of how transformer parameters influence data processing.