Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere

This paper investigates the continuous-time dynamics of self-attention when Rotary Position Embeddings (RoPE) are applied to queries and keys while values remain on the unit sphere. By modeling the layer index as time, the author treats token representations as an interacting particle system. The study focuses on the equilibrium states, spectral properties of the consensus kernel, and regional convergence rates, using a combination of differential geometry, spectral analysis, and numerical verification.

• Equilibria and Stability: The system admits consensus states as equilibria, but the presence of RoPE introduces "twisted" states and antipodal configurations that are absent in standard spherical attention. The author demonstrates that the natural interaction energy is not monotonic, meaning there is no universal Lyapunov function for the system.
• Spectral Analysis: On a resonant ring, the author derives an exact Bessel-aliasing spectrum for the consensus kernel. This spectrum reveals that the consensus gap depends non-monotonically on the energy distribution across frequency planes, and that certain modes are unreachable due to non-coprime frequency conditions.
• Convergence Rates: The paper establishes a kernel-generic positivity principle. It proves that closed hemispheres are forward-invariant and that tokens in non-obtuse configurations or strict open semicircles are guaranteed to converge to a consensus point with explicit, sharp exponential decay rates.

Understanding the dynamics of RoPE is critical because it is the standard positional encoding for modern Transformers. This work provides a rigorous mathematical foundation for how RoPE alters the "attention landscape," explaining why certain configurations lead to slower convergence or stable non-consensus states. By providing explicit bounds and spectral formulas, the paper offers researchers a way to predict the behavior of deep residual stacks without relying solely on empirical observation.