MXSENS: SENSITIVITY-AWARE MIXED-PRECISION QUANTIZATION FOR EFFICIENT LLM INFERENCE

Large Language Models (LLMs) are increasingly quantized to 4-bit precision to reduce memory and compute costs. However, this aggressive compression often leads to significant accuracy degradation due to systematic outliers—disproportionately large values in activations that distort quantization blocks. While existing solutions like rotation-based methods or software-managed mixed-precision help, they often introduce substantial overhead or are incompatible with modern hardware-accelerated microscaling formats like MXINT.

The authors introduce MXSens, a training-free quantization method designed to mitigate outlier-induced errors by assigning variable bitwidths (4, 6, or 8 bits) to weights and activations. Instead of using uniform precision, MXSens identifies the most sensitive columns and layers using Hessian-based metrics. It then allocates higher precision to these sensitive regions, naturally aligning with the block-wise structure of MXINT, which allows for efficient, hardware-native mixed-precision inference.

• Sensitivity-Guided Allocation: The authors demonstrate that quantization sensitivity is not uniform; it varies significantly across both layers and columns. By identifying these sensitive components, MXSens can selectively apply higher precision where it is most needed.
• Triplet Quantization: By using a 4-6-8 bit strategy, MXSens effectively captures extreme outliers (8-bit) and mild outliers (6-bit) while keeping the majority of the model at 4-bit precision. This provides a better accuracy-efficiency trade-off than binary (4/8-bit) or uniform quantization.
• Superior Performance: MXSens consistently outperforms state-of-the-art quantization baselines across various models, including LLaMA-2 and LLaMA-3. For example, in the W4A4KV4 setting, it achieves significantly lower perplexity on WikiText-2 compared to existing methods, all without requiring model fine-tuning or architectural changes.

As LLMs grow in size, deploying them efficiently on hardware becomes a primary bottleneck. MXSens provides a practical path to high-accuracy, low-bit inference that is compatible with emerging hardware standards like NVIDIA Blackwell. By avoiding the overhead of software-managed scaling and the complexity of model retraining, this approach makes high-performance, compressed LLMs more accessible for real-world deployment.