GRINQH: Graded Input-based Quantization Hierarchy for Efficient LLM Generation

Large Language Models (LLMs) are frequently bottlenecked by the memory wall—the disparity between high-speed on-chip computation and slower off-chip DRAM transfer speeds. While quantization is a standard solution, existing methods often treat the prefill (compute-bound) and decoding (memory-bound) stages identically. This paper asks whether decoupling these stages and applying dynamic, input-dependent precision can improve inference efficiency without sacrificing model accuracy.

The authors introduce GRINQH (Graded Input-based Quantization Hierarchy), a framework that uses activation magnitudes as a proxy for computational importance. During the decoding stage, GRINQH evaluates incoming activations to assign varying bit widths (0–8 bits) to weight channels. Channels associated with low-magnitude activations are assigned lower precision or skipped entirely, while high-magnitude 'outlier' channels are preserved at higher precision. To implement this, the authors developed a custom GPU kernel that uses a hierarchical bit-planar memory layout, allowing the system to fetch only the necessary bits from DRAM for each weight channel.

GRINQH consistently outperforms state-of-the-art fixed-precision and mixed-precision baselines (such as GPTQ and AWQ) at comparable bit widths. By effectively managing the trade-off between precision and memory traffic, the framework establishes a new Pareto frontier for LLM generation, enabling faster inference speeds at 2-, 3-, and 4-bit settings while maintaining higher accuracy than static methods. The authors demonstrate that this dynamic approach effectively mitigates the memory-bandwidth bottleneck, particularly in edge-computing environments where batch sizes are small.