ResearchPod Summary
This paper investigates how the leading eigenvectors of the Hessian matrix—which represent the directions of highest curvature in the loss landscape—evolve during neural network training. The authors seek to understand whether different optimization algorithms (SGD, Adam, and SAM) traverse the loss landscape in qualitatively different ways by tracking the stability and localization of these curvature directions.
To quantify the evolution of the Hessian's leading eigenvectors, the authors employ two primary metrics:
The authors track these metrics for multilayer perceptrons trained on MNIST and FashionMNIST, comparing the results against a random null model to isolate the effects of the optimization process itself.
The study reveals that optimization dynamics are highly optimizer-dependent. SGD exhibits a stabilizing effect, where leading curvature directions become increasingly fixed over time. In contrast, Adam causes continuous, strong reorganization of eigenvectors throughout the training process, effectively 'mixing' the curvature directions. Furthermore, the authors identify a striking localization phenomenon under Adam, where the most significant curvature directions become concentrated on a small subset of model parameters, whereas SGD maintains delocalized curvature. These differences suggest that Adam and SGD explore fundamentally different regions of the loss landscape, with Adam's dynamics potentially reflecting a more aggressive or exploratory search strategy.
[[RP_SECTION:optimizer-impact-on-curvature|Optimizer impact on curvature]]
Alex: [steady, analytical] Optimizers don't merely minimize loss — they actively reshape the geometry of the loss landscape as they run. Adam, specifically, induces persistent eigenvector mixing and parameter-specific localization of curvature. That's the headline result from a 2026 workshop paper on optimizer-dependent training dynamics by Marjankowska and colleagues.
Sam: [curious, leaning in] So the choice of optimizer fundamentally changes the curvature of the landscape as the model learns? I always thought of the Hessian as a static property of the model and data — a consequence of architecture and dataset, not of the update rule.
Alex: [measured] That's the standard view, and this paper pushes back on it directly. By tracking the leading Hessian eigenvectors throughout training, the authors show that SGD leads to stable curvature directions — those eigenvectors settle and stay put. Adam, by contrast, causes continuous, optimizer-driven reorganization of those same directions. The landscape isn't just being descended; it's being actively restructured by the update rule itself. [[RP_SECTION:measuring-eigenvector-displacement|Measuring eigenvector displacement]]
Sam: [processing] If the eigenvectors are constantly shifting under Adam, how do you distinguish that from noise? Stochastic gradients introduce variance everywhere — how do they isolate the optimizer's contribution?
Alex: [precise] They use a two-time correlation function to measure eigenvector displacement across training steps. Think of it like a compass needle for the loss landscape. Under SGD, that needle stabilizes — it shifts as the model learns, but it points in a consistent direction. Under Adam, the needle spins continuously, recalibrating throughout training in a way that's systematic, not stochastic. [[RP_SECTION:localization-and-parameter-structure|Localization and parameter structure]]
Sam: [probing] And what about localization? If the eigenvectors are rotating, does the spatial structure of curvature — which parameters actually carry it — also change?
Alex: [deliberate] That's the second major finding, and in some ways the more striking one. They measure it using the Inverse Participation Ratio, which quantifies how concentrated an eigenvector is across the parameter space. Under Adam, the leading eigenvectors become strongly localized on a tiny subset of parameters. Under SGD, curvature spreads across the network — delocalized, diffuse. So Adam is doing two things simultaneously: rotating the directions of maximum curvature, and concentrating that curvature onto a narrow set of weights.
Understanding Hessian eigenvector dynamics provides a deeper look into the 'black box' of neural network training. By showing that optimizers leave distinct geometric fingerprints on the loss landscape, this work offers a new diagnostic tool for researchers to evaluate how different training methods interact with model architecture. These insights could eventually inform the development of more efficient pruning techniques, better uncertainty quantification, and improved optimization strategies that leverage the specific ways different algorithms navigate curvature.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: [connecting] And those two effects are independent? The localization persists even as the eigenvectors rotate?
Alex: [confirming] That's the key point. The authors report that the IPR — the localization statistic — is stable even when the eigenvectors themselves are actively mixing. So you have this situation where the *which* is stable but the *where* is not. The curvature stays concentrated on roughly the same parameter subset, but the directions of that curvature keep reorganizing. It's a dissociation that the standard picture of loss landscape geometry doesn't really account for. [[RP_SECTION:implications-for-model-training|Implications for model training]]
Sam: [thinking through implications] That has direct consequences for pruning and uncertainty quantification. If Adam concentrates curvature on a narrow parameter subset, then the effective dimensionality of the problem is much lower than the raw parameter count suggests — and a pruning strategy that ignores optimizer choice is probably making implicit assumptions it shouldn't.
Alex: [measured] Exactly. And it cuts the other way too. If you're doing something like Laplace approximation for uncertainty, where you're inverting or approximating the Hessian, the localization structure Adam induces means your approximation quality depends heavily on which optimizer trained the model. Two networks with identical architectures and datasets but different optimizers will have fundamentally different curvature geometry at convergence. [[RP_SECTION:scaling-and-future-research|Scaling and future research]]
Sam: [grounding the scope] We should be clear about what the paper is and isn't. These results come from small-scale perceptrons. A careful referee would immediately ask whether this localization and mixing behavior survives into the regime of large transformers — models where the Hessian is never computed explicitly and the parameter count is orders of magnitude larger.
Alex: [balanced, not dismissive] That's the right caveat. The authors are working in a setting where they can actually track the leading Hessian eigenvectors, which is computationally tractable only at small scale. Whether the same dissociation — stable localization, unstable eigenvector directions — appears in billion-parameter models is an open question. The mechanism is plausible at scale, but the empirical evidence doesn't extend there yet.
Sam: [landing the synthesis] So the core contribution is diagnostic rather than prescriptive at this point. They've developed the measurement framework — the two-time correlation function, the IPR — and used it to show that optimizer identity leaves a distinct geometric signature on the trained model. SGD and Adam don't just converge differently; they produce structurally different loss landscapes.
Alex: [steady] And that reframing matters. If the optimizer is an architect of curvature geometry, not just a descent strategy, then choices we've been making on empirical grounds — Adam for transformers, SGD for vision — may have geometric consequences we haven't been accounting for. The next step is understanding whether that geometry can be controlled deliberately: whether you can steer localization by design, or whether it's an emergent property of the adaptive moment estimation that's difficult to decouple.
Sam: [reflective] We've been treating optimizers as black boxes that move down a hill. This paper makes the case that they're also reshaping the hill as they go — and that the shape they leave behind is measurable, structured, and optimizer-specific. That's a meaningful shift in how to think about training dynamics.
Alex: [closing] Thanks for listening to ResearchPod.