How the Hessian-Spectrum of Neural Networks Depends on Data

This paper investigates the relationship between the geometry of the training data and the curvature (sharpness) of the loss landscape in deep neural networks. While the Hessian matrix is central to understanding optimization dynamics and generalization, prior theoretical work often relied on overly simplified settings. The authors aim to provide a rigorous characterization of the Hessian spectrum for linear networks of arbitrary depth and width, specifically examining how data properties influence the sharpness of the final solution.

The authors utilize the generalized Gauss-Newton (GGN) approximation of the Hessian, which is closely related to the Neural Tangent Kernel (NTK). By assuming that the network layers are "strongly balanced"—meaning singular values are aligned across layers—they derive the eigenvalues of the Hessian as a function of the singular values of the weight matrices and the input data. They systematically analyze the effects of these assumptions, testing their theoretical predictions against empirical results on standard datasets like MNIST, FashionMNIST, and CIFAR10.

The study identifies that the sharpness of the solution is directly linked to the distribution of labels in the training set. Specifically, datasets with a dominant class (high maximum label proportion) lead to sharper solutions. The authors also show that sharpness increases with network depth under certain conditions. While their primary results assume isotropic features and balanced weights, they demonstrate that these predictions remain qualitatively robust even when incorporating nonlinearities like Tanh activations or using standard Kaiming initialization.

Understanding the Hessian spectrum is critical for designing better optimization algorithms, improving generalization, and developing efficient pruning or quantization techniques. By providing a clear link between data geometry and loss landscape curvature, this work challenges existing intuitions—such as the belief that sharpness necessarily increases with dataset size—and offers a more precise framework for analyzing why certain datasets are easier to learn than others.