Marcelina Marjankowska, Valerio Modugno, Paolo Barucca
5 min
This paper investigates how the leading eigenvectors of the Hessian matrix—which represent the directions of highest curvature in the loss landscape—evolve during neural network training. The authors seek to understand whether different optimization algorithms (SGD, Adam, and SAM) traverse the loss landscape in qualitatively different ways by tracking the stability and localization of these curvature directions.
To quantify the evolution of the Hessian's leading eigenvectors, the authors employ two primary metrics:
The authors track these metrics for multilayer perceptrons trained on MNIST and FashionMNIST, comparing the results against a random null model to isolate the effects of the optimization process itself.
The study reveals that optimization dynamics are highly optimizer-dependent. SGD exhibits a stabilizing effect, where leading curvature directions become increasingly fixed over time. In contrast, Adam causes continuous, strong reorganization of eigenvectors throughout the training process, effectively 'mixing' the curvature directions. Furthermore, the authors identify a striking localization phenomenon under Adam, where the most significant curvature directions become concentrated on a small subset of model parameters, whereas SGD maintains delocalized curvature. These differences suggest that Adam and SGD explore fundamentally different regions of the loss landscape, with Adam's dynamics potentially reflecting a more aggressive or exploratory search strategy.
Understanding Hessian eigenvector dynamics provides a deeper look into the 'black box' of neural network training. By showing that optimizers leave distinct geometric fingerprints on the loss landscape, this work offers a new diagnostic tool for researchers to evaluate how different training methods interact with model architecture. These insights could eventually inform the development of more efficient pruning techniques, better uncertainty quantification, and improved optimization strategies that leverage the specific ways different algorithms navigate curvature.
Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors identify which parameters generate that curvature. In this work, we study how the leading Hessian eigenvectors evolve during training and how they affect the learning trajectories. We track the training dynamics of multilayer perceptrons on a classification problem and measure eigenvector dynamics through two complementary statistics: (i) displacement over time, inspired by analyses of glassy systems, and (ii) localization via the inverse participation ratio. The metrics are compared against a random null model of the Hessian induced by the architecture. Our results reveal clear optimizer-dependent behaviour. SGD leads to progressively more stable leading curvature directions, while Adam exhibits substantially stronger reorganization of eigenvectors throughout training. We also observe a localization phenomenon under Adam, where a small subset of parameters contributes disproportionately to the leading curvature directions. These results suggest that Hessian eigenvector dynamics capture key differences in optimizer behaviour and the resulting training trajectories.
Alex: [confirming] That's the key point. The authors report that the IPR — the localization statistic — is stable even when the eigenvectors themselves are actively mixing. So you have this situation where the *which* is stable but the *where* is not. The curvature stays concentrated on roughly the same parameter subset, but the directions of that curvature keep reorganizing. It's a dissociation that the standard picture of loss landscape geometry doesn't really account for. [[RP_SECTION:implications-for-model-training|Implications for model training]]
Sam: [thinking through implications] That has direct consequences for pruning and uncertainty quantification. If Adam concentrates curvature on a narrow parameter subset, then the effective dimensionality of the problem is much lower than the raw parameter count suggests — and a pruning strategy that ignores optimizer choice is probably making implicit assumptions it shouldn't.
Alex: [measured] Exactly. And it cuts the other way too. If you're doing something like Laplace approximation for uncertainty, where you're inverting or approximating the Hessian, the localization structure Adam induces means your approximation quality depends heavily on which optimizer trained the model. Two networks with identical architectures and datasets but different optimizers will have fundamentally different curvature geometry at convergence. [[RP_SECTION:scaling-and-future-research|Scaling and future research]]
Sam: [grounding the scope] We should be clear about what the paper is and isn't. These results come from small-scale perceptrons. A careful referee would immediately ask whether this localization and mixing behavior survives into the regime of large transformers — models where the Hessian is never computed explicitly and the parameter count is orders of magnitude larger.
Alex: [balanced, not dismissive] That's the right caveat. The authors are working in a setting where they can actually track the leading Hessian eigenvectors, which is computationally tractable only at small scale. Whether the same dissociation — stable localization, unstable eigenvector directions — appears in billion-parameter models is an open question. The mechanism is plausible at scale, but the empirical evidence doesn't extend there yet.
Sam: [landing the synthesis] So the core contribution is diagnostic rather than prescriptive at this point. They've developed the measurement framework — the two-time correlation function, the IPR — and used it to show that optimizer identity leaves a distinct geometric signature on the trained model. SGD and Adam don't just converge differently; they produce structurally different loss landscapes.
Alex: [steady] And that reframing matters. If the optimizer is an architect of curvature geometry, not just a descent strategy, then choices we've been making on empirical grounds — Adam for transformers, SGD for vision — may have geometric consequences we haven't been accounting for. The next step is understanding whether that geometry can be controlled deliberately: whether you can steer localization by design, or whether it's an emergent property of the adaptive moment estimation that's difficult to decouple.
Sam: [reflective] We've been treating optimizers as black boxes that move down a hill. This paper makes the case that they're also reshaping the hill as they go — and that the shape they leave behind is measurable, structured, and optimizer-specific. That's a meaningful shift in how to think about training dynamics.
Alex: [closing] Thanks for listening to ResearchPod.