Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable

Pretrained language models often exhibit high variance in their ability to perform needle-in-a-haystack (NIAH) retrieval, a phenomenon where models trained with identical architectures and data but different random seeds show drastically different success rates. This "seed lottery" means that practitioners cannot reliably predict whether a model will successfully retrieve information, even if its perplexity and loss curves appear healthy. Standard Rotary Positional Encoding (RoPE) contributes to this issue because its frequencies are aperiodic, causing positional phases to scramble and wander erratically over long distances.

Möbius RoPE addresses this by replacing standard RoPE frequencies with an anti-periodic frequency ladder. By setting the rotation frequency to an odd multiple of pi, the positional holonomy becomes -1. This creates a closed-form "dipole" structure where the two ends of the context window are deterministically coupled with a sign flip. This geometric scaffold acts as a long-range channel, providing the model with a stable, predictable structure that facilitates the formation of retrieval circuits during training.

In experiments across 160M and 410M parameter models, the hybrid Möbius approach (applying these frequencies to 25% of attention heads) consistently matches the perplexity of standard RoPE while significantly reducing retrieval variance. At a context length of 512, the worst-performing seed for the Möbius model achieved 86% accuracy, compared to just 14% for the standard RoPE baseline. Matched control experiments confirm that this effect is specifically tied to the anti-periodic boundary condition, as periodic or aperiodic ladders fail to replicate the same reliability floor.

This research demonstrates that retrieval capability is not purely a function of model scale or training data, but is heavily influenced by the underlying positional geometry. By using a simple, zero-cost frequency swap, researchers can convert a stochastic capability into a reliable one. This provides a form of "pretraining insurance," ensuring that models possess robust in-context retrieval skills without requiring additional parameters, FLOPs, or architectural complexity.