Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?

Traditional neural network pruning and interpretability methods often assume that neurons with larger absolute weight values are inherently more important to the model's decision-making process. This paper investigates the validity of this assumption by rigorously testing whether 'high-weight' neurons are truly the most critical for maintaining classification accuracy.

The authors evaluate neuron importance through three distinct experimental lenses on CIFAR-10 and Mini-ImageNet models:

1. Overlap Analysis: They compare the set of top-weighted neurons against a set of 'accuracy-impacting' neurons (identified by individual ablation) to see if the two groups coincide.
2. Perturbation Testing: They apply various perturbations (ablation, equidistant, and proportional) to neurons across different weight intervals and measure the resulting degradation in model accuracy.
3. Ablation-Retraining: They remove neurons from specific weight intervals and retrain the remaining network to determine if the model can recover performance, thereby testing the functional necessity of those specific neurons.

The results challenge the weight-importance equivalence. The authors find that the top 10% of high-weight neurons overlap with the most important neurons by only 25% at best. Furthermore, while ablating the top 10% of high-weight neurons causes significant accuracy drops (10-20%) that the model cannot recover from, a third of these high-weight neurons actually have minimal impact on performance. Conversely, some low-weight neurons are found to be just as critical as mid-range high-weight neurons, demonstrating that importance is nonlinear and not strictly tied to magnitude.

This research provides a more nuanced understanding of how neural networks store and process information. By proving that weight magnitude is an unreliable indicator of importance, the study suggests that current pruning and compression strategies—which often prioritize removing low-weight neurons—may be suboptimal. These insights can guide the development of more effective model compression, encryption, and interpretability techniques that focus on functional contribution rather than simple weight statistics.