Ziqing Qiao, Yinuo Xu, Chaojun Xiao, Zhou Su, Zihan Zhou, Yingfa Chen, Xiaoyue Xu, Xu Han, Zhiyuan Liu
4 min
Abstract
Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent sequence mixers. However, how these efficient modules shape model capabilities remains poorly understood. To address this gap, we conduct a systematic analysis across hybrid architectures from three perspectives: scaling behavior, mechanism analysis, and architecture design. First, from a scaling perspective, we find that efficient-attention design primarily affects how fast long-context capability emerges, while different hybrids eventually converge to comparable long-context performance under sufficient training. Second, mechanistically, we show that long-range retrieval is mainly carried by full attention, whereas efficient attention shapes its optimization trajectory. This explains a counter-intuitive phenomenon we call Large-Window Laziness: larger SWA windows can delay the formation of retrieval heads in full-attention layers. Third, guided by this mechanism, we show that applying NoPE to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance with negligible impact on short-context performance.
Alex: And that's why these models eventually catch up, given enough training time? The laziness isn't permanent—it just slows things down?
Sam: Right. Given enough training data and time, even the lazier models get there. But the paper argues we can do better than just waiting. The fix they propose involves removing something called "positional encoding" from the full-attention layers specifically.
Alex: Wait—how does removing positional information help? I'd have thought knowing the order of words was essential.
Sam: It's a reasonable instinct. Normally, positional encoding is like giving every word in a sentence a numbered tag—word one, word two, word three, and so on. The model uses those tags to understand sequence and order. But when you remove those tags from the full-attention layers, something useful happens: the model can no longer lean on position as a shortcut. It's forced to understand words by their meaning and how they relate to each other, regardless of where they sit in the document. It's the difference between finding a book by its shelf number versus finding it because you know what it's about.
Alex: So without the positional crutch, the model has to build a more flexible, content-based way of searching through long documents—and it builds that skill earlier in training.
Sam: That's the argument. And the paper reports this leads to meaningful improvements on long-context tasks without hurting performance on shorter text. The key insight is that the bottleneck isn't the model's raw capacity—it's the path the training process takes to get there. By adjusting what the full-attention layers are forced to rely on, you change which skills develop first.
Alex: So the lesson for anyone designing these systems is: don't just ask whether the architecture can handle long documents. Ask whether the training process is actually pushing it to learn how.
Sam: That's well put. The structure of a model doesn't just determine what it can do—it shapes what it bothers to learn. And those are two very different things.
Alex: Thanks for listening to ResearchPod.