Cascaded Multi-Granularity Pruning for On-Device LLM Inference in Industrial IoT

Deploying Large Language Models (LLMs) on Industrial Internet of Things (IIoT) edge devices is hindered by extreme memory and latency constraints. Existing structured pruning methods often fail at high compression ratios because they rely on one-shot importance estimation, which ignores how pruning one component (e.g., a layer) shifts the importance of others. To address this, the authors propose a cascaded multi-granularity pruning framework that removes components in a coarse-to-fine order: layers, then attention heads, then feed-forward (FFN) channels. Between these stages, the authors employ lightweight Low-Rank Adaptation (LoRA) to recover performance and re-estimate the importance of remaining components.

The authors provide an information-theoretic justification for this coarse-to-fine ordering by modeling the LLM as a Markov information processing chain. By applying the Data Processing Inequality (DPI), they argue that removing coarser structures first minimizes information loss. Furthermore, they introduce the Structural Independence Assumption (SIA) to determine if a model architecture is suitable for per-component pruning. They demonstrate that while MHA+GELU architectures satisfy the SIA, modern designs like GQA+SwiGLU violate it, leading to predictable performance instability.

The framework was tested on bearing fault diagnosis tasks using models ranging from 88M to 6.25B parameters. On MHA+GELU architectures, the method achieved 13.8x compression with 83.82% accuracy, outperforming the strongest baseline by 3.70 percentage points. Conversely, the authors show that applying this framework to GQA+SwiGLU architectures results in a ~74pp accuracy collapse, validating the predictive power of the SIA. Deployed on an industrial slewing bearing platform using NVIDIA DGX Spark, the compressed models reduced inference latency by 67.2% and peak memory usage by 62.5%.