Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention

Modern transformers rely on in-context learning to adapt to new tasks, but their reliance on softmax attention creates a bottleneck. Softmax attention is nonparametric, meaning it stores past tokens in a growing key-value (KV) cache. As the sequence length increases, the memory and compute requirements grow quadratically, making it impossible to maintain long-term memories over the lifetime of an AI agent on fixed hardware.

The authors propose reframing attention as an online learning algorithm. In this view, the transformer generates a stream of self-supervised key-value pairs at inference time. The attention mechanism acts as a regressor, attempting to learn a mapping from keys to values. While traditional softmax attention uses a nonparametric approach (similar to kernel regression) that requires storing all past data, the authors argue for parametric attention. Parametric attention uses a fixed-size set of weights to represent this mapping, allowing the model to summarize and compress past experiences rather than just storing raw tokens.

Parametric attention mechanisms—such as linear attention, state-space models, and test-time training layers—offer a path toward lifelong learning. By maintaining a constant memory footprint, these models can theoretically process unbounded sequences. The authors highlight that the most promising approach involves 'test-time training,' where the model uses gradient descent to update its internal parameters based on incoming data. This allows the model to evolve its understanding of tasks over time, rather than simply recalling specific past observations.

Despite the potential of parametric attention, the field faces significant open questions. The authors emphasize that simply minimizing the error on past key-value pairs is insufficient; the model must learn to generalize. Future research needs to address how to design better online objectives, how to incorporate explicit regularization to prevent overfitting to recent tokens, and how to balance the need for fast, parallelizable updates with the capacity to store complex, long-term information.