Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
8 min
Abstract
Weight-activation quantization can reduce the memory and compute cost of neural-network inference, but recent methods often require expensive optimization and architecture-specific propagation of basis transforms across layers. We introduce MACaQ, an MXFP4 W4A4 post-training quantization method that treats every linear layer independently. MACaQ applies a per-channel scale and a block-Hadamard transform online to each layer's activations, while absorbing the corresponding inverse transforms into its weights. Retaining these transforms online avoids cross-layer dependencies and enables a simple streaming implementation of the quantization process, in which only one layer must reside in memory at a time. To improve over round-to-nearest quantization, we introduce Flip-Refinement, a Hessian-aware coordinate-descent procedure that greedily moves individual FP4 codes to adjacent representable values. Despite executing the activation transforms at inference time, MACaQ adds only 2.3% end-to-end overhead over pure MXFP4 on Qwen3-32B in vLLM. MACaQ reaches lower perplexity than DuQuant++ and MR-GPTQ, and already matches their best perplexity on Llama-3-8B in 11 times less quantization time. We further apply MACaQ without architecture-specific modifications to an image diffusion model, illustrating the generality of its layer-local design and show quantization of a 122B parameter model in 64 minutes, using only 32.5GB GPU RAM.
Sam: Make that concrete for one layer. What happens to its inputs, and what gets prepared in advance?
Alex: First, each input channel gets its own scale, chosen to balance activation and weight magnitudes. Then a Hadamard transform, a fixed mixing operation, spreads concentrated energy within groups of thirty-two channels. The corresponding inverse adjustment is absorbed into the weights before those weights are quantized.
Sam: Why mix only small groups? Wouldn’t mixing across the entire input spread those large values even further?
Alex: The storage format gives each small group a shared scale. Mixing across groups could spread one outlier into several groups that must each accommodate it. MACaQ aligns the mixing groups with the format’s scaling groups instead.
Sam: That prepares the numbers for compression. But ordinary rounding still throws information away. What does the method do beyond rounding to the nearest value?
Alex: It uses calibration inputs, examples passed through the model, to estimate which weight errors matter for the layer’s output. That information forms a Hessian, an input-derived matrix describing how weight errors affect reconstruction. The method first uses it to choose each weight group’s shared scale.
Sam: And the new refinement step uses that same information to reconsider the rounded weights?
Alex: Yes. Flip-Refinement asks whether moving a weight to the next representable value, upward or downward, would reduce the reconstruction objective. For each output row, it takes the best improving move, updates its calculations efficiently, and repeats.
Sam: That sounds narrower than searching freely over all compressed weights. It’s a local search starting from a cheap initial answer.
Alex: That’s the intended trade-off. The group scales stay fixed during refinement, and each move considers only adjacent values. Unlike GPTQ’s sequential weight assignment, this procedure starts with rounded weights and revisits their codes.
Sam: Let’s separate the promises now. Does faster preparation mean giving up quality, or do the comparisons show both improving?
Alex: On the tested language models, MACaQ has lower perplexity than the reported competing quantization methods. Perplexity measures language-prediction error; lower is better. In the Llama comparison, it matches the strongest baselines’ best perplexity with eleven times less quantization time, measured on the same hardware.
Sam: That’s a preparation result, not a guarantee that generated answers stay equally good. Did they test what happens when models actually generate?
Alex: They tested instruction following and mathematical reasoning as well. MACaQ was competitive, but not uniformly the best compressed method on every task. On the dense Qwen model’s mathematics test, another method scored higher, and MACaQ remained below the original full-precision model.
Sam: So better perplexity doesn’t mean every downstream score improves. What about the runtime bill for keeping those transformations online?
Alex: On the largest dense Qwen model they measured, the extra decoding overhead was 2.3 percent compared with the same compressed implementation without scaling and mixing. That’s not a comparison against full precision. Their compressed implementation was also faster than full precision.
Sam: A low overhead on a large model could conceal a less attractive result on smaller ones. Does that happen?
Alex: It does: overhead reaches about eight percent for the smallest tested model. The transforms depend on input width, while matrix multiplication also grows with output width. Larger layers therefore spread the extra work across more computation, although actual timing also depends on memory traffic and kernel implementation.
Sam: You also mentioned equipment requirements. What’s the clearest demonstration that independent layers change what can be prepared?
Alex: They quantized a model with 122 billion parameters whose full-precision weights could not fit on their target graphics processor. Loading layers as needed, preparation took sixty-four minutes. Peak device memory during that process was about thirty-three gigabytes.
Sam: That memory figure is for preparing the model, not necessarily for running the finished model?
Alex: Correct. The finished model required substantially more memory, but still fit on the same processor. And there’s a scope restriction: for mixture-of-experts models, which route computation through selected expert modules, they quantized only the expert layers.
Sam: Does the architecture-independence claim go beyond language models, or is it mainly an implementation argument?
Alex: They also applied the method to an image diffusion model, FLUX Klein, without architecture-specific modifications. Its images were closer to the original high-precision model than an official alternative compressed version across the reported fidelity measures. That supports portability, though fidelity to the original isn’t a complete measure of image quality.
Sam: What’s the main limitation you’d carry into a decision about using this?
Alex: Layer independence also rules out cross-layer error compensation. Each layer improves its own reconstruction rather than coordinating corrections across the network. I’d also treat the speed measurements as hardware-specific: these experiments use one reported GPU configuration, not a broad hardware survey.
Sam: Given that balance, who should read the full document, and where should they start?
Alex: Researchers building low-precision inference systems should read it closely. Start with Quantization Procedure for the streaming design, then the quality-versus-preparation-time comparison. Read End-to-End Speed and Limitations before assuming the trade-off fits your model.
Sam: And for researchers who only need the central idea?
Alex: Saving every runtime operation is not always the best overall bargain. Here, a small runtime cost buys simpler preparation, lower preparation memory, and competitive compressed-model quality.
Sam: Keep that trade-off in mind when choosing what to optimize.