KronQ: LLM Quantization via Kronecker-Factored Hessian

Existing post-training quantization (PTQ) methods, such as GPTQ, rely exclusively on input activation statistics to determine how to compress model weights. This approach implicitly assumes that all output channels are equally important to the model's performance. The authors investigate whether incorporating output-side information—specifically the gradient covariance—can improve quantization accuracy and stability, particularly for large language models at ultra-low bit-widths.

KronQ leverages the Kronecker-factored Hessian approximation, which factorizes the full weight Hessian into the product of input activation covariance and output gradient covariance. The authors introduce two primary innovations:

1. Bidirectional Incoherence Processing: Extending existing rotation-based techniques, they use gradient covariance to rotate the output dimension, reducing weight magnitude variance across both input and output dimensions.
2. Joint Hessian Sensitivity Metric: They derive a new metric for mixed-precision allocation that uses the traces of both activation and gradient Hessians. This allows the model to distinguish the sensitivity of different sublayers (like Query, Key, and Value projections) that share the same input statistics but have different output roles.

By accounting for output-side Hessian information, KronQ significantly stabilizes the quantization process. In experiments on the LLaMA-2 and LLaMA-3 model families, KronQ consistently achieves lower perplexity than state-of-the-art baselines like GPTQ and GPTAQ. The most dramatic improvement occurs at 2-bit weight-only quantization on LLaMA-3-70B, where KronQ achieves a perplexity of 7.93, whereas competing methods produce degenerate results with perplexities exceeding 2000.