C-PTQ: Fisher-weighted Channel-wise Sensitivity for Post-training Quantization of MLLMs

Multimodal Large Language Models (MLLMs) are computationally expensive, making post-training quantization (PTQ) essential for deployment. However, standard quantization often degrades performance because it fails to account for the varying importance of different channels. Existing methods typically use modality- or token-level metrics to guide scaling, which often misaligns with the actual task-specific loss. The authors ask: Can we directly inject channel-wise importance into the scaling process to better preserve task accuracy?

The authors propose C-PTQ, a unified channel-wise PTQ method. They model the impact of quantization on the model's loss function using a second-order Taylor expansion. By assuming the model is near a local optimum, they simplify the loss perturbation to a quadratic form involving the Hessian matrix. Since the full Hessian is computationally prohibitive, they approximate it using the empirical Fisher Information Matrix. By further applying a diagonal approximation, they derive a Fisher-weighted objective that assigns a specific sensitivity weight to each channel. This weight is then integrated into the channel-wise scaling search, ensuring that the most task-critical channels receive the most protection during the quantization process.

C-PTQ consistently outperforms existing state-of-the-art PTQ methods across various MLLMs, including Qwen2.5VL, InternVL2, and LLaVA-OV. In both weight-only and weight-activation quantization settings, C-PTQ demonstrates superior performance on a wide range of benchmarks, such as MMMU and TextVQA. The results show that by aligning channel-wise scaling with gradient-based task sensitivity, the model maintains higher fidelity without the need for auxiliary modules like LoRA, preserving both high performance and computational efficiency.