Reliability Scaling Laws for Quantized Large Language Models

Research Overview

This paper investigates the reliability of quantized Large Language Models (LLMs) by moving beyond standard accuracy metrics. While quantization is widely used to reduce the memory and latency of LLMs, its impact on model trustworthiness—specifically uncertainty estimation, calibration, and robustness to input noise—has remained largely unexplored. The authors introduce a comprehensive evaluation framework to assess how these reliability dimensions scale as a function of the total number of model bits (the product of parameter count and bit-precision).

The authors evaluate LLMs across a range of sizes (1B to 70B parameters) and quantization levels (2, 3, 4, and 8 bits) using six different quantization methods. The reliability framework consists of three pillars:

• Uncertainty: Measuring the model's confidence in its predictions using entropy and log-likelihood.
• Calibration: Assessing whether the model's confidence scores align with its actual accuracy using the Brier Score.
• Robustness: Testing performance against 15 types of semantically-preserving input perturbations, including character-level typos, slang, emoji insertion, and leetspeak.

Key Findings

The study reveals a critical divergence between standard performance and reliability. While accuracy consistently improves as the total bit budget increases, reliability metrics exhibit a non-monotonic, "U-shaped" or peaked behavior. Specifically, 4-bit quantized models consistently achieve the best balance between efficiency and reliability. Furthermore, the authors find that quantization can actually enhance a model's robustness to natural, non-adversarial input perturbations compared to their full-precision counterparts.