Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models

Post-training quantization (PTQ) is a popular method for compressing deep neural networks, but it often leads to significant performance drops at low bit-widths (e.g., 2-bit or 4-bit). The authors investigate whether the accuracy of low-bit PTQ is limited not only by the quantization process itself, but also by the inherent sensitivity of the full-precision (FP) model to quantization errors. They hypothesize that models residing in flatter loss basins are more robust to these errors and seek a way to steer FP models toward such regions before quantization occurs.

The authors propose Efficient Tuning Before Quantization (ETBQ), a pre-conditioning stage performed after standard training but before PTQ. Unlike Quantization-Aware Training (QAT), which trains a fake-quantized model, ETBQ keeps the model in full precision. During this stage, the model is fine-tuned while being subjected to perturbations sampled from the specific error distributions of weight and activation quantization.

To ensure the model moves toward a flatter, more robust region, the authors employ two key mechanisms:

• Weight Quantization Noise (WQN): Models weight quantization errors as channel-wise Gaussian distributions and uses temporal differencing to prevent drift and encourage flatter solutions.
• Activation Quantization Noise (AQN): Models activation quantization errors as per-tensor Gaussian distributions, using a salience-aware stochastic mask to improve robustness to activation-side perturbations.

Experiments across CIFAR-100, Tiny-ImageNet, ImageNet, and Cityscapes demonstrate that ETBQ consistently improves performance for low-bit PTQ. For example, under W2A4 settings, ETBQ improved top-1 accuracy on Tiny-ImageNet by 2.14% and mIoU on Cityscapes by 5.80% compared to naive PTQ. Hessian analysis confirms that ETBQ shifts the model toward a flatter loss landscape, characterized by smaller eigenvalues and a reduced Hessian trace, which directly correlates with higher tolerance to quantization-induced errors.