Disentangling the Expressivity of RoPE

Rotary position embeddings (RoPE) have become the de facto position encoding scheme in transformer-based language models. Despite their widespread adoption, theoretical explanations for their success remain divided. Expressivity studies view RoPE through the lens of periodicity and modular predicates, whereas mechanistic and long-context analyses emphasize positional anchors and local offsets. This paper reconciles these two accounts within a fully uniform, finite-precision transformer framework, demonstrating that the theoretical behavior of RoPE depends entirely on whether its rotational frequencies are periodic or incommensurate.

The authors formalize the distinction between two types of RoPE schedules under finite-precision arithmetic. When every rotary component is periodic—meaning its rotation matrix repeats after a fixed number of steps—the realized positional encoding corresponds to a finite table lookup indexed by the position modulo that period. In contrast, conventional RoPE utilizes angular frequencies that are incommensurate with 2pi, meaning the computed rotations never truly repeat. This non-periodic schedule yields a precision-dependent bounded simulation of fixed-offset look-back operators rather than an all-length modular characterization.

The core theoretical contribution establishes that component-periodic RoPE transformers (SMAT[RoPE_P]) recognize exactly the class of languages defined by past temporal logic with modular predicates (LTL[P, MOD]). Absolute sinusoidal positional encodings with periodic schedules are shown to share this exact expressivity boundary. Furthermore, the paper investigates the formal properties of LTL[P, MOD], proving that it is incomparable with the star-free languages (LTL[S]), as it can capture modular periodic patterns like (ab)* but cannot perform symbol-counting tasks such as PARITY.