Memory for Large Language Models

Memory in large language models has transitioned from an implicit byproduct of standard transformer computation to a first-class architectural design dimension. Historically, models relied on the transformer's attention mechanism—a form of transient, content-addressable working memory—to manage context. However, the quadratic scaling of attention and the ephemeral nature of these states have driven researchers to develop explicit, persistent, and adaptive memory systems. This paper provides a comprehensive survey of these advancements, organizing them into a structured taxonomy to help researchers navigate the increasingly fragmented landscape of memory-centric LLM design.

The authors introduce a three-axis framework to characterize memory mechanisms:

• Representation: Distinguishes between implicit memory (tightly coupled with the forward computation graph, such as KV caches or recurrent states) and explicit memory (distinct storage components with defined read/write semantics, such as dedicated parameter modules or lookup tables).
• Update Dynamics: Categorizes memory based on when updates occur—offline (during training) versus online (during inference)—and further refines this with specific update rules, such as gradient-based optimization, state-transition updates, or signal-gated writing.
• Persistence: Differentiates between short-term memory (transient, typically bounded by a context window or cache budget) and long-term memory (designed to retain information over extended sequences or across sessions via compression or persistent storage).

By consolidating disparate approaches—ranging from state-space models and sparse attention to test-time training and mixture-of-experts—into a single framework, this survey enables a more principled comparison of architectural trade-offs. It highlights the shift toward models that can dynamically evolve during deployment, offering a roadmap for future research in scalable, adaptive language modeling. The authors also identify critical open challenges, including the need for a unified theory of memory, hardware-algorithm co-design, and more robust multi-dimensional evaluation methodologies.