HiKonv: High Throughput Quantized Convolution With Novel Bit-wise Management and Computation

Modern hardware units, such as CPUs and DSPs, are typically designed for high-precision arithmetic (e.g., 32-bit or 64-bit). When these units are used to process low-bitwidth quantized Deep Neural Networks (DNNs), the majority of the hardware's bit-capacity remains underutilized. This paper asks: how can we systematically reconfigure existing high-precision arithmetic units to maximize throughput for quantized convolutions of varying bitwidths?

To address this, the authors propose HiKonv, a unified solution that treats high-precision multipliers as parallel engines for low-bitwidth data. The core technique involves 'input slicing' and 'data packing,' where multiple low-bitwidth operands are concatenated into a single high-precision word. By carefully inserting 'guard bits' between these packed operands, the authors ensure that the resulting partial products do not overlap or overflow during computation. The method provides a theoretical framework to determine the optimal number of operations a given multiplier can perform per cycle based on the input bitwidth and the hardware's native bit-capacity.

HiKonv demonstrates that significant throughput gains are possible without modifying the underlying hardware. On a standard CPU, the authors achieved up to a 3.17x latency improvement for quantized convolution layers. When applied to an FPGA-based implementation of the UltraNet model, HiKonv outperformed the state-of-the-art by 2.37x in throughput and 2.61x in DSP efficiency. The authors show that as the target quantization bitwidth decreases, the number of parallel operations per cycle increases, making the approach particularly effective for highly quantized models like Binary Neural Networks (BNNs).

This work provides a bridge between high-precision hardware and the growing demand for efficient, low-bitwidth DNN inference. By offering a mathematically grounded strategy for bit-wise management, HiKonv allows researchers to extract maximum performance from existing silicon. It not only improves the deployment of current models but also provides a template for future hardware-software co-design, where arithmetic units can be optimized specifically for the flexible bit-packing patterns HiKonv enables.