Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

This paper investigates how the leading eigenvectors of the Hessian matrix—which represent the directions of highest curvature in the loss landscape—evolve during neural network training. The authors seek to understand whether different optimization algorithms (SGD, Adam, and SAM) traverse the loss landscape in qualitatively different ways by tracking the stability and localization of these curvature directions.

To quantify the evolution of the Hessian's leading eigenvectors, the authors employ two primary metrics:

1. Displacement: A two-time correlation function borrowed from the study of glassy systems, which measures how much an eigenvector rotates or drifts over time.
2. Inverse Participation Ratio (IPR): A measure from statistical physics used to determine if an eigenvector is 'localized' (concentrated on a small subset of parameters) or 'delocalized' (spread uniformly across the network).

The authors track these metrics for multilayer perceptrons trained on MNIST and FashionMNIST, comparing the results against a random null model to isolate the effects of the optimization process itself.

The study reveals that optimization dynamics are highly optimizer-dependent. SGD exhibits a stabilizing effect, where leading curvature directions become increasingly fixed over time. In contrast, Adam causes continuous, strong reorganization of eigenvectors throughout the training process, effectively 'mixing' the curvature directions. Furthermore, the authors identify a striking localization phenomenon under Adam, where the most significant curvature directions become concentrated on a small subset of model parameters, whereas SGD maintains delocalized curvature. These differences suggest that Adam and SGD explore fundamentally different regions of the loss landscape, with Adam's dynamics potentially reflecting a more aggressive or exploratory search strategy.