Rubin Wei, Jiaqi Cao, Jiarui Wang, Junming Zhang, Qipeng Guo, Bowen Zhou, Zhouhan Lin
5 min
Standard decoder-only language models entangle reasoning and long-term memory within the same set of parameters, making it difficult to scale memory capacity without increasing the entire model size. This paper investigates whether decoupling long-term memory into a separate, modular, and pretrained parametric component allows for more efficient scaling and domain adaptation.
The authors introduce a scalable architecture where a frozen base language model is paired with a swappable, parametric memory module. During pretraining, the memory module is trained to align its output distribution with a non-parametric k-nearest neighbor (kNN) retriever, which provides rich supervision by capturing diverse potential continuations. To manage the computational burden of constructing kNN distributions at the scale of 207B tokens, the authors developed a distributed Faiss pipeline that utilizes embedding compression, index sharding, and parallel GPU search. The final model prediction is an interpolation of the base model's output and the memory module's output.
The study demonstrates that allocating parameters to a dedicated memory module yields a superior parameter-performance tradeoff compared to scaling the backbone model. For instance, a Pythia-410M model paired with a 6.9B parameter memory outperformed a standalone Pythia-12B model on 17 benchmarks while using 39% fewer total parameters. Furthermore, domain-specific memories (e.g., for biology, law, and finance) consistently improved performance across various backbone scales and even showed successful transfer across different model families and vocabularies.
This work provides a practical path for building more efficient and modular language models. By treating memory as an independent, scalable component, researchers can improve model performance on knowledge-intensive tasks without the prohibitive costs of full-model retraining or the risks of catastrophic forgetting associated with standard fine-tuning. This modularity also enables the reuse of specialized domain memories across different base models.
Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.
Alex: So they're not just proposing a new model architecture — they're building the infrastructure to make it work at scale.
Sam: That's the real contribution. And the results support the approach. A reasoning backbone with around 410 million parameters, paired with a memory module roughly seventeen times larger, outperforms a single unified model that's bigger than both combined. The implication is that scaling memory separately is more efficient than simply making one large model bigger.
Alex: Does this modular approach hold up across different subject areas?
Sam: That's one of the more practical aspects of the design. Because the reasoning backbone never changes, you can attach different memory modules for different domains — one trained on legal documents, another on biomedical research, another on financial data. The model doesn't need to relearn how to reason; it just gets a new set of specialised knowledge to draw from. And crucially, adding a law module doesn't risk erasing what the model already knows about biology or programming.
Alex: It's a bit like a consultant who has strong analytical skills and just picks up a new briefing document before each project.
Sam: That's a fair analogy. The underlying capability stays constant; only the knowledge being applied changes. What this paper ultimately argues is that the field may have been conflating two things that are better kept separate — the ability to reason, and the store of facts to reason about. Keeping them distinct opens up a more flexible and cost-effective path for keeping models current over time.
Alex: That's a genuinely useful reframe for thinking about how these systems are built. Thanks for walking us through it, and thanks to everyone listening to ResearchPod.