Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
5 min
Abstract
Test-time scaling is a powerful approach to obtain better reasoning in large language models, but it becomes memory-bottlenecked during long-horizon decoding, as the KV-cache grows. KV-cache quantization can help improve this, but current methods are evaluated under prefill-like settings and errors behave differently under autoregressive decoding. We show that in the latter regime, quantization errors accumulate across timesteps, driven primarily by incorrect token scales. We introduce KVarN, a calibration-free KV-cache quantizer that applies a Hadamard rotation followed by a dual-scaling variance normalization across both axes of the K and V matrices. We find that this combination fixes outlying token-scale errors and substantially reduces error accumulation over existing baselines. KVarN establishes a new state-of-theart for KV-cache quantization on generative benchmarks, including MATH500, AIME24 and HumanEval, at 2-bit precision. A vLLM implementation of the KVarN method is available at https://github.com/huawei-csl/KVarN
Sam: Exactly. The researchers found that most existing compression methods don't account for these scaling errors at all. They focus on preserving the direction of the data, but they let the magnitudes drift. That's the gap KVarN is designed to close.
Alex: So how does it actually fix the problem?
Sam: KVarN uses two techniques working together. The first is called a "Hadamard rotation." Don't worry about the name — the idea is straightforward. Before compressing the data, the system scrambles the information across many different channels simultaneously. This prevents any single piece of information from being an extreme outlier that throws off the whole compression step. It's like spreading a heavy load across a wide bridge instead of putting all the weight on one weak plank.
Alex: So the rotation makes the data more uniform before you compress it.
Sam: Precisely. And the second technique is where the real work happens. It's called "dual-scaling variance normalization." The word "variance" here just means how spread out the values are. If some values are enormous and others are tiny, rounding them all to the same simplified format will distort the big ones badly and the small ones less so. What this technique does is look at the data and rescale it — both across different time steps and across different internal channels — so that everything lands within a consistent, predictable range before the rounding happens.
Alex: It's like a sound engineer normalizing every track in a mix so nothing is peaking into the red.
Sam: That's a very accurate way to put it. By keeping everything within a predictable range, the rounding process becomes much more stable. The combination of scrambling the data first and then normalizing its spread is what stops those small errors from accumulating over long sequences.
Alex: One thing I want to ask about — you mentioned this method works "on the fly," without needing to be trained on specific data first. Why does that matter?
Sam: It matters a lot for flexibility. Some compression methods need to be calibrated — you feed them examples of the kind of text the model will encounter, and they learn how to compress that specific type of data. KVarN doesn't need that. It dynamically adjusts the scale as it generates each token, which means it works just as well on a legal document as it does on a math problem, without any special preparation.
Alex: And does all this extra calculation slow things down?
Sam: That's the part worth paying attention to. The authors measured this carefully after implementing KVarN in a real inference system. The additional normalization step adds only a fraction of a percent to the total processing time — essentially negligible. The stability gains come at almost no cost in speed.
Alex: So the bottleneck for long-horizon reasoning isn't just the model's intelligence — it's the way we compress its memory.
Sam: That's the key insight. By fixing these magnitude errors in the memory buffer, the model can maintain accurate reasoning over thousands of steps. And because it works at very low memory precision, it opens the door to running high-capability reasoning models on consumer hardware, where memory is often the most constrained resource.
Alex: That's a clear and useful breakdown of the problem and the fix. Thanks for walking through the mechanics, Sam.
Sam: It's a good example of how identifying one specific mathematical flaw — not the model's architecture, not its training data, just the way numbers are rounded in memory — can lead to a meaningful improvement in real-world performance. Thanks for listening to ResearchPod.