ResearchPod Summary
As transformer models scale to longer contexts, the quadratic computational complexity of causal self-attention becomes a primary bottleneck. While many post hoc linearization techniques exist—converting full-attention models into linear-time architectures without full retraining—these methods often combine multiple interventions (e.g., LoRA, sliding windows, distillation) simultaneously. This paper seeks to isolate the fundamental design question: which linear state update mechanism best approximates the behavior of pretrained softmax attention?
To isolate the approximation capacity of different linear mechanisms, the authors evaluate them in a strict frozen-backbone regime, where the original transformer weights are kept fixed and only the new linear replacement parameters are trained. They derive a first-order Taylor approximation of softmax attention, demonstrating that it relies on key-dependent, rank-1 orthogonal projections. They compare three primary linear attention families: kernelized attention (Hedgehog), gated accumulation (GLA), and delta-style updates (Gated Delta Networks). Finally, they introduce structural interventions—specifically sink tokens, sliding-window attention (SWA), and short convolutions—to compensate for identified approximation errors.
This work provides a principled, analysis-driven framework for transformer linearization. By identifying that delta-style updates are theoretically closer to softmax than standard gated linear attention, it offers a more efficient and effective path for converting massive pretrained models (such as LLaMA and Qwen up to 32B parameters) into linear-time architectures without the need for expensive, full-scale retraining.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.