Co-LMLM: Continuous-Query Limited Memory Language Models

Traditional Large Language Models (LLMs) store factual knowledge implicitly within their parameters, which makes them prone to hallucinations, difficult to update, and challenging to attribute. Limited Memory Language Models (LMLMs) attempt to solve this by externalizing knowledge into an external database. Previous LMLM designs relied on relational databases and explicit, decoded textual queries, which limited the scope of externalizable knowledge to structured Wikipedia-style tuples.

This paper introduces Continuous-Query LMLM (CO-LMLM), which replaces structured relational queries with continuous vector queries. During generation, the model emits a special token that triggers a dense retrieval from a knowledge base (KB) containing vector keys and textual values. The model is trained jointly on next-token prediction and a contrastive objective that aligns the model's hidden states with the retrieved factual spans, allowing for more flexible and scalable knowledge retrieval.

CO-LMLM demonstrates significant improvements over both standard LLMs and previous relational LMLMs. At the 360M parameter scale, CO-LMLM achieves lower perplexity than standard models trained on 40 times more data. Furthermore, its factual precision, as verified by the SimpleQA benchmark, is competitive with much larger models like gpt-4o-mini and superior to Claude Sonnet 4.5. The authors also show that the model generalizes well beyond Wikipedia, maintaining high performance when trained on broader datasets like FineWeb-Edu.

CO-LMLM provides a path toward LLMs that are both highly performant and inherently controllable. By separating factual knowledge from the model's weights, it enables direct editing and unlearning of information without requiring full model retraining or compromising general linguistic capabilities. This architecture effectively bridges the gap between the high performance of parametric models and the transparency and editability of retrieval-augmented systems.