Tommaso Cerruti, Tim Rieder, George Rowlands, Lingfeng Jin, Imanol Schlag
5 min
Abstract
Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.
Alex: They developed a technique called Cross-Layer Value Routing. The idea is to create a shared pathway—think of it like a main highway running through the entire model—where information from earlier layers can be passed directly to later ones, bypassing the telephone-game problem entirely.
Sam: And what exactly travels along that highway?
Alex: This is where the finding gets interesting. The researchers first tried routing the error signal—the difference between what the model expected and what it actually found. That seemed logical, but it didn't improve performance. What actually worked was routing the core information itself, not just the correction signal. It's the difference between whispering a full summary of a book to the next librarian, versus just whispering a note about what was filed incorrectly.
Sam: So the more complete signal was more useful than the precise correction. Why would that be?
Alex: Because the correction only tells you what went wrong. The full value tells you what the information actually was. The next layer can do much more with a complete picture than with a description of a gap.
Sam: And did the researchers build in any safeguards to stop this pathway from causing problems?
Alex: They did. The routing starts at zero—the model begins by ignoring the extra pathway entirely, and only learns to use it if it finds a genuinely useful signal to pass along. That way, it can't accidentally make things worse. The gains are modest but consistent: giving the model a direct, well-aligned pathway helps it maintain memory integrity across layers, particularly in cases where the standard memory hasn't fully captured the data.
Sam: And alongside the routing, the paper also discusses gating. How does that fit in?
Alex: Gating works like a set of adjustable knobs on the memory system. The model learns to tune these to decide how much of a new memory to write down, how much to erase, and how quickly to let older information fade. Rather than blindly storing everything, the model becomes selective—it learns to hold onto what matters and let go of what doesn't.
Sam: So between the delta-rule updates, the cross-layer routing, and the gating, the researchers are building a much more disciplined memory system.
Alex: That's the right way to think about it. And perhaps the broader contribution of this work is that it brings clarity to a field that had become quite cluttered. There were many variants of these models, each with slightly different designs, and it wasn't always obvious how they related to each other. This paper organizes them into a coherent family, makes the design choices explicit, and shows which ones actually matter. That kind of legibility is genuinely useful for anyone building on this work going forward.
Sam: It sounds like the paper is as much about understanding the landscape as it is about any single result.
Alex: Exactly. Good research doesn't always mean finding a dramatic breakthrough. Sometimes it means making a complicated space navigable—so that the next set of researchers can move through it more confidently. Thanks for listening to ResearchPod.