PolyQ: Codesigning End-to-End Quantization Framework for Scalable Edge CPU LLM Inference

As LLMs move to edge devices, CPUs have become the default inference target due to their ubiquity and power efficiency. However, existing quantization methods struggle to balance the need for fine-grained, activation-aware bit allocation with the strict execution requirements of CPU architectures. The authors seek to bridge the gap between coarse, layer-wise quantization and inefficient, fine-grained mixed-precision methods by co-designing a quantization strategy with a compiler-based execution framework.

PolyQ introduces a two-stage pipeline to achieve fractional-bit deployment on CPUs:

1. Quantization Strategy: The framework uses an activation-aware error proxy to assign bit-widths (from a {2, 3, 4, 8, 16}-bit palette) to individual channels. It treats this as a waterfilling problem, where bits are allocated to channels that yield the highest reduction in reconstruction error per bit, ensuring the model stays within a user-specified fractional average-bit budget.
2. Compiler Co-design: To address the execution overhead of heterogeneous bit-widths, the compiler clusters channels into bit-homogeneous blocks. It generates layer-specific kernels compatible with SIMD and LUT-based instructions and propagates channel permutations across operators. This ensures that layout regularization—the process of reordering data to fit hardware kernels—is performed at compile time rather than during runtime.

PolyQ demonstrates that fractional-bit CPU deployment is both practical and efficient. Across models like Llama2-13B and Qwen3-32B, the framework provides stable quality scaling and improves perplexity by up to 32.1% compared to prior methods at a 3-bit target. On various CPU targets, the compiler-driven layout regularization reduces activation reorder traffic by over 70%, and the end-to-end energy overhead remains below 2% relative to optimized LUT-based back-ends. This allows for precise memory-budget fitting without sacrificing inference speed.