Junyu Chen, Jungang Li, Jing Xiong, Wenjie Wang, Qingyao Yang, He Xiao, Zhen Li, Taiqiang Wu, Mengzhao Chen, Zhen Peng, Chaofan Tao, Long Shi, Hongxia Yang, Ngai Wong
8 min
Abstract
Large language model (LLM) inference is often bounded by memory footprint and memory bandwidth in resource-constrained deployments, making quantization a fundamental technique for efficient serving. While post-training quantization (PTQ) maintains high fidelity at 4-bit, it deteriorates at 2-3 bits. Fundamentally, existing methods enforce a shape-invariant quantization grid (e.g., the fixed uniform intervals of UINT2) for each group, severely restricting the feasible set for error minimization. To address this, we propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients, and iteratively refines them using approximate second-order information while progressively compensating quantization errors to minimize output discrepancy. In the 2-bit regime, BPDQ enables serving Qwen2.5-72B on a single RTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit). Moreover, we provide theoretical analysis showing that the variable grid expands the feasible set, and that the quantization process consistently aligns with the optimization objective in Hessian-induced geometry. Code: github.com/KingdalfGoodman/BPDQ.
Alex: So the bit-planes give the basic binary building blocks, and the scalars stretch or shrink them differently for each group of weights. That breaks the fixed pattern—does it naturally handle the big outlier values that are crucial for the model?
Sam: Yes, precisely. Outliers—those unusually large or small weights that carry key information—get captured in the higher bit-planes, which BPDQ preserves by design. The group-specific scalars then fine-tune the spacing around them. This aligns with the optimization goal from calibration data, where the best quantization minimizes how much the model's outputs change.
Alex: Okay, that sets up a good starting grid. But to refine it further, do they just tweak those scalars iteratively?
Sam: They do—through an iterative process alternating two steps. First, for each column, they try all possible combinations of bit-planes—up to 2^k options, but k is small like 3 for 2-bit, so it's manageable—and pick the best under the error metric. Then they refit the scalars group-wide in closed form, adding a delta correction to keep error propagation consistent across columns. This back-and-forth expands the grid's flexibility.
Alex: You've sketched the back-and-forth refinement. Walk me through one cycle—starting with how they pick those bit combinations for a column.
Sam: They process one column of weights at a time within the group. For each spot in that column, they test all possible on-off patterns across the k bit-planes—like flipping switches for the 1s, 2s, 4s layers to make a number from 0 up to the max. They compute what value each pattern gives after scaling by the current group's knobs, then choose the pattern closest to the original weight in a simple squared difference. They follow with an error carry-over step to the next column, keeping things consistent. After updating bit-planes across the full group, they refit the scalar coefficients from scratch, solving a weighted least-squares problem row by row.
Alex: That sounds precise. But refitting changes things—how do they avoid messing up the error tracking?
Sam: They add a delta correction right after. The new quantized block differs slightly from the old one before refit, so they calculate the difference in the propagation coordinates and add it back to the error state for that group. Across 10 cycles picking the best by group error, it hones the grid without drifting.
Alex: The alternation plus correction keeps everything aligned to that output-preserving goal. It's about three times slower than GPTQ, but holds a 72B model near full-precision accuracy on reasoning tasks.
Sam: Yes, and that stems directly from the variable grid's freedom, letting the solver get closer to the ideal than fixed grids. It outperforms baselines notably on reasoning tasks.
Alex: You've mentioned it lags vector methods slightly. What do the comparisons show against those, and why does preserving outliers matter so much?
Sam: Vector quantization works by assigning each small group of weights to the nearest entry in a shared codebook—like picking the best pre-made recipe from a cookbook for a batch of ingredients. This gives high fidelity but takes much longer to compute, around 40 times GPTQ's quantization time. BPDQ comes close in accuracy on reasoning tasks while staying efficient, because its variable grid naturally protects outliers—those key large values in the model's signals.
Alex: So outliers are like the standout signals that carry the model's smarts. Does the paper confirm that links to better performance?
Sam: Yes, they measured activation outliers using ratios of max to median values per layer. GPTQ at 2 bits suppresses them notably, while BPDQ and vector methods retain them well. This preservation correlates with stronger results on math and reasoning benchmarks.
Alex: Interesting link. And for real-world use, like handling long inputs—does it hold up there?
Sam: They tested long-context tasks like retrieval and summarization. At 3 or 4 bits, BPDQ stays robust, matching full precision closely. At 2 bits, it's tougher especially for retrieval, but still outperforms fixed-grid methods overall.
Alex: That rounds out the picture—efficient quantization that guards key features without massive slowdowns. A solid advance for fitting big models on everyday hardware. But the authors mention some limits—what are the main trade-offs?
Sam: A key one is a small fidelity gap compared to vector quantization, which takes far longer to set up. BPDQ's refinement runs about three times slower than GPTQ due to those cycles, and it depends on quality calibration data from real inputs. Still, the paper positions it as a balanced step for consumer GPUs. The binary bit-planes also lend themselves to specialized hardware, swapping complex math for additions to save energy, and support mixed precision naturally.
Alex: Makes sense—efficiency matters more than perfection for on-device stuff. So in essence, by ditching fixed grids for flexible ones built from simple binary pieces, BPDQ opens the door to huge models on regular devices while keeping reasoning solid.
Sam: Precisely. The paper shows it relaxes those shape constraints, enabling high-fidelity 2-bit runs for 72 billion parameter models on single consumer GPUs. It points to on-device AI as a real possibility.
Alex: That's a clear, grounded contribution to squeezing big AI into everyday life. Thanks for breaking it down, Sam—and thanks for listening to ResearchPod.