CARVE: CONTENT-AWARE RECURRENT WITH VALUE EFFICIENCY FOR CHUNK-PARALLEL LINEAR ATTENTION

Recurrent models in the delta-rule family offer a memory-efficient alternative to Transformers, but existing state-of-the-art architectures (like GDN-2) suffer from three coupled defects: they are "memory-blind" (forgetting without seeing what is stored), they waste parameters on high-dimensional write gates, and their gating structure prevents the use of efficient parallel chunk-solvers. This paper asks if these limitations can be resolved simultaneously without sacrificing hardware efficiency.

The authors introduce CARVE (Content-Aware Recurrent with Value Efficiency). The core innovation is a structural constraint: all gating operations are restricted to the key axis. This constraint is mathematically proven to be both necessary and sufficient to restore the WY-form triangular chunk solver, which allows for efficient parallel training. Within this framework, CARVE implements two key improvements:

1. Content-Aware Erase: Instead of reading the entire state matrix from memory, CARVE reuses the recurrent output tensor—already computed for the forward pass—to form a chunk-level content signal. This allows the erase gate to "see" what is stored in memory at zero additional memory-access cost.
2. Scalar Write Gate: The model replaces high-dimensional per-value write projections with a single scalar per head, significantly reducing parameter counts without losing associative capacity.

CARVE was evaluated at the 1.3B-parameter scale on 100B tokens. It outperformed the GDN-2 baseline on WikiText perplexity, common-sense reasoning benchmarks, and RULER retrieval probes. Crucially, these gains were achieved with 19% fewer mixer parameters, 13% lower peak memory, and negligible throughput overhead. The authors provide six formal theorems covering stability, gradient flow, and expressivity to ground these empirical results.