ATMA: Length-Invariant Language Modeling via Polar Attention and Gated-Delta Compression Memory

Modern Large Language Models (LLMs) often suffer from performance collapse when processing sequences longer than their training context. This is largely attributed to softmax-based attention mechanisms, where probability mass dilutes as the sequence grows, and a structural tension between sliding-window attention (which lacks long-range recall) and full-context attention (which suffers from perplexity explosion). The author investigates how to achieve length-invariant modeling that preserves both high-fidelity retrieval and low perplexity at extreme context lengths.

To address these limitations, the paper introduces ATMA, a hybrid architecture that integrates a three-channel attention mechanism. The core innovation is Polar Attention, which factorizes attention into:

1. A count-blind, unit-vector direction channel: Isolates 'what' was matched regardless of the number of matches.
2. A bounded magnitude channel: Uses the participation ratio of matches to quantify 'how much' was matched, mapped through a saturating function to prevent activation shift.
3. A long-term recurrent compression memory: Utilizes a gated-delta fast-weights rule (inspired by the Titans model) to store historical context as a lossy gist.

The model employs a 3:1 ratio of gated convolutional layers to Polar Attention layers, ensuring computational efficiency while maintaining global mixing capabilities.

Through a 120-run factorial ablation sweep, the author demonstrates that neither the Polar Attention core nor the recurrent memory is sufficient in isolation. However, their combination enables monotonic perplexity reduction and robust long-range retrieval. Specifically, ATMA maintains induction needle-in-a-haystack retrieval accuracy above 90% out to 64K tokens (32x the training length), significantly outperforming softmax-based baselines that collapse at extreme lengths. Furthermore, the model exhibits self-stabilizing properties in its memory state, preventing the norm explosion typically seen in linear attention systems.