Simplifying Neural Networks During Training

Modern deep neural networks are often heavily overparameterized, raising questions about whether all layers are necessary for task performance. This paper investigates whether the training dynamics of a network can be monitored to identify the transition point between feature extraction and classification, allowing for the removal of redundant trailing layers during the training process itself.

The authors propose Neural Network Simplification (NNS), a framework that combines insights from Neural Collapse (NC) and the Tunnel Effect (TE). The core of the method is the Inverse Fisher Criterion (IFC), a metric that tracks the ratio of within-class variance to between-class variance. By monitoring the IFC across layers and epochs on a small, class-balanced subset of data (the tunnel set), the authors identify when a layer shifts from building expressive representations to performing task-specific compression. Once this split point is identified and stabilized, the trailing layers are discarded, a lightweight classification head is attached, and the reduced model continues training.

The study demonstrates that the IFC provides a stable and consistent signal for identifying the extractor-classifier split across various architectures, including MLPs, VGGs, and ResNets. By applying NNS, the authors show that it is possible to significantly reduce the number of parameters in a network while maintaining accuracy comparable to the original, full-sized model. This approach avoids the need for post-hoc pruning by making structural decisions dynamically during the optimization phase.

This work provides a practical, data-driven method for reducing the computational and environmental footprint of deep learning models. By operationalizing the theoretical concepts of Neural Collapse and the Tunnel Effect, the authors offer a principled way to design more efficient training pipelines that favor smaller, more compact architectures without sacrificing predictive power.