Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs

Large Language Models (LLMs) are often compressed using Singular Value Decomposition (SVD) to reduce their memory and compute footprint. While standard SVD provides a mathematically optimal low-rank approximation, it is agnostic to the functional importance of specific weights. Existing methods either rely solely on forward-pass activation statistics or require expensive end-to-end fine-tuning. This paper asks: can we integrate backward-signal influence metrics into a layer-local SVD framework to achieve better performance-preserving compression without the high cost of end-to-end optimization?

The authors introduce Activation- and Influence-Aware Ranks (AIR). The framework begins with the activation-aware optimal solution provided by SVD-LLM. It then incorporates a backward-signal influence metric (such as LRP, Weight×Gradient, or Fisher information) into the objective function. Because element-wise weighted low-rank approximation is generally intractable, the authors derive a closed-form Alternating Least Squares (ALS) sweep that iterates through ranks to redistribute approximation error away from high-influence weights. This process is layer-local, computationally efficient, and provides a monotone-descent guarantee.

AIR consistently outperforms existing SVD-based baselines across various models, including LLaMA, Mistral, and Vicuna. At a 60% parameter retention rate, AIR achieves significant perplexity improvements over SVD-LLM(W) (e.g., >18% on LLaMA-7B). The authors demonstrate that AIR is complementary to other techniques; for instance, AIR+LoRA fine-tuning outperforms end-to-end methods like ACIP. Furthermore, the method is highly efficient, requiring minimal calibration data and providing tangible gains in peak memory usage and per-token latency during inference.