RELATIVE POSITIONS GENERALIZE, ABSOLUTE POSITIONS MEMORIZE: AN IMPLICIT-BIAS ACCOUNT OF LENGTH GENERALIZATION IN ATTENTION

The Length Generalization Gap

Length generalization—the ability of a model to perform correctly on sequences longer than those encountered during training—is a persistent challenge in transformer research. A well-known empirical regularity is that models using relative positional encodings (like RoPE) often extrapolate successfully, while those using learned absolute positional encodings (APE) typically fail. Previous explanations for this gap have focused on expressivity, asking whether a generalizing solution is even possible. This paper shifts the focus to optimization, investigating why gradient descent selects a generalizing solution in some cases but not others.

The Role of Implicit Bias

By analyzing a minimal fixed-offset retrieval task, the authors demonstrate that the gap is governed by the implicit bias of the attention mechanism. For rotary encodings (RoPE), the attention logit is a function of the relative offset between query and key. This creates an exact equivariance: the rule learned during training (e.g., "attend to the token at offset -K") is structurally identical at any sequence length. Consequently, the model applies the same selection rule verbatim to unseen, longer sequences.

In contrast, learned absolute encodings (APE) do not possess this property. Because the model only encounters a finite range of absolute positions during training, the positional embeddings for indices outside that range remain untrained. The model effectively "pins" its attention to a fixed absolute position within the training range. When presented with a longer sequence, this pinned position no longer corresponds to the target offset, causing accuracy to collapse to chance levels.

The Carrier Mechanism and Accuracy Decay

Beyond the binary success or failure of generalization, the authors characterize the learned rotary rule as a low-rank "carrier" kernel. This kernel aligns the query and key representations to the target offset. The paper derives an "attention-dilution law" to explain why accuracy gracefully decays as sequences grow longer: as the context length increases, the relative weight of the target token is diluted by the growing number of non-target positions. This model-based prediction is confirmed empirically across various seeds and offsets, showing that the mechanism is robust and not merely an artifact of a single-layer architecture.