Denoised Variance-Based Pruning with Optimal Brain Bias Compensation

Vision Transformers (ViTs) and modern convolutional networks achieve remarkable accuracy but impose massive computational overhead that hinders deployment on resource-constrained edge devices. Structured pruning reduces this overhead by removing entire neurons or filters, but traditional magnitude-based methods often cause severe accuracy degradation, while existing variance-based pruning (VBP) suffers from statistical noise in finite-sample activation covariance and inadequate compensation. This paper introduces Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB2C) to enable robust, training-free structured pruning that retains high accuracy without expensive fine-tuning.

The proposed framework addresses the two primary limitations of prior variance-based pruning through a unified statistical approach. First, it uses random matrix theory, specifically the Marchenko-Pastur distribution, to filter noise from the activation covariance spectrum, producing reliable denoised variance scores for neuron selection. Second, it integrates mean-shift compensation directly into the Optimal Brain Compression (OBC) objective, proving that the layer-wise Hessian reduces exactly to the activation covariance matrix. This mathematical unification enables an optimal, closed-form update of the remaining weights using the identical statistics gathered for neuron selection.

Extensive evaluations across DeiT, Swin, and ConvNeXt architectures on ImageNet-1K demonstrate that DVBP + OB2C substantially outperforms existing training-free pruning baselines, especially at aggressive compression ratios. At a 50% MLP pruning ratio, the method retains over 90% of the original Top-1 accuracy on Small and Base variants. It consistently surpasses prior variance-based pruning by significant margins—outperforming VBP by up to 29.46% on ConvNeXt-T and 7.33% on Swin-S—without requiring any post-pruning retraining or fine-tuning.