DART: DECODED ATTENTION OVER RECURRENT STATES FOR EFFICIENT LONG-CONTEXT SEQUENCE MODELING

Modern language models often choose between the quadratic compute cost of Transformers (which offer precise retrieval) and the linear-time efficiency of recurrent models like Mamba-2 (which use compact states). This paper asks whether a single memory representation can simultaneously support recurrent compression and attention-style retrieval, effectively bridging the gap between these two architectures.

The authors introduce DART (Decoded Attention over Recurrent sTates), which builds upon the State Space Duality (SSD) framework of Mamba-2. While Mamba-2 decodes values from its compressed state, it does not decode keys. DART addresses this by retaining chunk state contributions—the compressed representations of sequence segments—as chunk state memories. It then uses token-conditioned projections to extract keys and values from these memories, performing State-Memory Attention (SMA) to retrieve relevant information. This retrieval is integrated into the model via a gated residual connection, allowing the model to learn when to rely on the recurrent output versus the retrieved memory.

DART significantly improves associative recall and long-context retrieval performance compared to the base Mamba-2 architecture while maintaining general language-modeling quality. By operating over chunked memories rather than individual tokens, DART achieves substantial memory savings during inference; for instance, with a chunk size of 256 and a state size of 128, it reduces the length-dependent inference cache by 75% compared to a matched attention baseline. The authors demonstrate that this approach is computationally practical by implementing the SMA mechanism using a custom FlashAttention-style kernel.

This work provides a path to high-performance long-context modeling that avoids the quadratic memory growth of standard Transformers. By demonstrating that recurrent states can be treated as searchable associative memories, DART offers a more efficient alternative to interleaved hybrid models, which typically require stacking separate attention and recurrent layers to achieve similar recall capabilities.