Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo
8 min
Abstract
We present the first systematic study of Massive activations (MAs) in layer-interleaved HLA LLMs and uncover two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP). As full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology of full attention LLMs. We establish the recurrence of this organization across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models spanning 1.2B to 397B total parameters. Controlled pretraining of GDN-based hybrids at scales up to 1.3B shows that both morphologies emerge early and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating their layerwise organization, whereas removing GDN gates yields comparatively modest amplification. Mechanistically, our systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology characteristic of full attention LLMs. Our code is available at https://github.com/StartluxLabs/Massive-Activations-HLA.
Sam: That's a good way to put it. The paper traces this to a specific three-step cycle the authors call the "write-sink-cancel lifecycle." Here's how it works: first, a layer just before a full attention block writes an unusually large value into the model's running signal — that's the "write." Then, during full attention, that position acts as what's called an "attention sink," meaning other positions in the sequence tend to focus on it disproportionately — that's the "sink." Finally, a layer just after cancels that large value out with an opposite signal — that's the "cancel."
Alex: So the spike appears, does its job during the attention step, and then gets cleaned up afterward. Why does the timing of that cleanup matter so much?
Sam: Because if the cancellation happens quickly and locally, you get a sharp spike that rises and falls in just a few layers. But if the full attention layers are densely packed, the cleanup from one cycle overlaps with the buildup of the next — and instead of sharp spikes, you get those persistent plateaus. The authors show that at the extreme, when a model uses full attention everywhere, those plateaus smooth out entirely into the stable pattern you'd see in a standard non-hybrid model.
Alex: So the hybrid architecture is essentially finding a middle ground between two extremes — and the spikes and plateaus are the visible signature of that balancing act.
Sam: That's a precise way to put it. Now, one question you'd naturally ask is: does this only happen in one specific model, or is it a general property of hybrid architectures?
Alex: Right — it could just be a quirk of one particular design choice.
Sam: The paper addresses this directly by testing across twelve different publicly available model families, ranging from small configurations up to systems with nearly four hundred billion parameters. Across different sequence mixers and training approaches, those pre-attention spikes and plateaus remained consistently aligned with the full attention layers.
Alex: Even when models were fine-tuned for specific tasks rather than just trained on raw text?
Sam: Yes. The paper compared base models with their instruction-tuned versions — models further trained to follow human directions — and found the structural locations of the spikes stayed the same. The exact numerical size of the spikes shifted, but their positions didn't.
Alex: So the architecture acts like a permanent mold. The training data changes the intensity, but the shape is baked in by the design.
Sam: And the domain tests support that. Across five different text domains, the physical positions of the spikes were stable while only their magnitudes varied with the content.
Alex: Did the team also look at when these patterns first appear during training? Like, do they emerge gradually or are they there from the start?
Sam: They ran controlled pretraining experiments at smaller scales and watched the structures develop from the very beginning. The spikes were already visible after training on just one billion tokens — which is early in the process — and grew more pronounced as training continued.
Alex: So this isn't something that sneaks in late. It's an inherent part of how these networks learn to organize information from the beginning.
Sam: The evidence points that way. The team also tested whether adding output gates — essentially valves that control how much signal passes from one layer to the next — could suppress the spikes. Adding a gate to the full attention layer noticeably reduced the spike amplitude, but small pre-attention peaks still persisted throughout training.
Alex: Which suggests the network has a genuine functional need for some level of elevated activation right before those full attention steps, regardless of how much you try to dampen it.
Sam: That's the implication. The team also ran a controlled experiment where they took a smaller model and moved the single full attention layer to three different positions — early, middle, and late in the network. Placing it early produced only a weak bump. Placing it late created the largest spike.
Alex: So the deeper the full attention layer sits, the more the preceding layers have had time to build up that anticipatory pressure.
Sam: Exactly. More layers of local processing before the bottleneck means more accumulated signal waiting to be coordinated.
Alex: Okay, so pulling back — what does the paper ultimately say this all means for building better models?
Sam: The authors suggest that by mapping out this write-sink-cancel lifecycle precisely, researchers now have a clear framework for understanding where numerical instability comes from in hybrid models. Future architectures could be designed to regulate the timing of that cancellation step more carefully — potentially eliminating memory bottlenecks during inference and enabling models that handle very long contexts smoothly without running into hidden numerical problems.
Alex: So instead of being surprised by these spikes, future systems could be designed to manage them deliberately.
Sam: That's the practical direction the paper points toward. Though it's worth noting what the study doesn't establish: it carefully maps where the spikes occur and how they form, but it doesn't provide a definitive account of exactly what computational job these massive activations are performing when the model actually generates text. That causal link remains an open question.
Alex: So we have a detailed map of the terrain, but not yet a complete explanation of why the terrain is shaped the way it is.
Sam: That's a fair summary. The map itself is useful — knowing where the bottlenecks are is the first step toward designing around them. And that's what makes this study a meaningful contribution to understanding how hybrid language models actually work under the hood.
Alex: Thanks for walking through all of that, Sam. And thanks to everyone listening to ResearchPod.