Revisiting the Volume Hypothesis

Modern deep neural networks are often heavily over-parameterized, yet they generalize well to unseen data. A long-standing debate seeks to explain this: is it the implicit bias of stochastic gradient descent (SGD) that pushes models toward simple, generalizing solutions, or is it an inherent property of the network architecture itself? The volume hypothesis suggests the latter, positing that regions of parameter space corresponding to high generalization accuracy are simply much larger (occupy more volume) than those that generalize poorly. Consequently, random sampling would eventually find these good solutions, and SGD is just one way to reach them.

Previous empirical studies have yielded contradictory results, with some showing that random sampling performs poorly compared to SGD, while others suggest that high-entropy (large volume) regions naturally align with good generalization. This paper argues that these discrepancies arise from different training data regimes.

The authors use the Replica Exchange Wang-Landau (REWL) algorithm, a technique from statistical physics, to estimate the joint density of states over training and test accuracies. By treating the network parameters as a system of binary weights, they can map the volume of parameter space corresponding to specific performance levels. This allows them to compare the "typical" generalization performance of a random model (the peak of the density) against the performance achieved by SGD across varying dataset sizes (30, 300, and 600 samples).

The study reveals that the generalization gap between random sampling and SGD is highly sensitive to the amount of training data. In small-data regimes, SGD significantly outperforms random sampling, indicating that the optimization process is essential for navigating the complex landscape to find rare, high-performing solutions. However, as the training dataset grows, the density of states concentrates, and the performance of the typical random solution approaches that of the SGD-trained model. This suggests that architectural bias and optimization bias are complementary: optimization is the primary driver when data is scarce, while architectural volume effects become the dominant factor as data constraints increase.