Sai Niranjan Ramachandran, Suvrit Sra
5 min
Abstract
We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.
Sam: Exactly. The practical upshot is a diagnostic signal — a way to forecast the onset of generalization before it happens. You could imagine a learning rate scheduler that doesn't operate on heuristics or fixed epoch counts, but instead responds to that divergence signal. Adjust hyperparameters when the system is approaching a transition, rather than after the fact.
Alex: Okay, but let me push on the foundations here. The percolation framing assumes these merging components are macroscopic — that the phase transition is real and not just a finite-size artifact. How solid is that assumption?
Sam: That's the central open question the authors themselves flag. The thermodynamic limit — what happens to this structure as the number of parameters grows without bound — is unresolved. If the merging components are small relative to the network size, the apparent discontinuity might smooth out at scale. Grokking might not be a sharp transition in large models; it might be a gradual crossover that just looks sharp at the scales we currently train at.
Alex: Which would significantly change how much weight you'd put on the topological interpretation.
Sam: It would. The authors point to empirical evidence from transformers — that the cascade structure holds up as model complexity increases — but that's not a proof. It's a suggestive pattern. The gap between the finite-size observations and a rigorous thermodynamic theory is real, and a careful referee would push hard on it.
Alex: So the load-bearing result is the geometric cascade in the variance signal, and the open question is whether that cascade survives to the scale where it would actually matter for frontier models.
Sam: That's a fair summary. What the paper establishes clearly is the mechanism at tractable scales and the diagnostic signal it produces. What it doesn't establish — and doesn't claim to — is that the same sharp transitions persist in the regime where most practitioners are actually working.
Alex: Still, as a framework it's doing something useful. It moves the conversation away from "add more data, add more compute" and toward understanding the internal structural dynamics of the network. Even if the thermodynamic limit is fuzzy, having a principled account of why grokking happens — and a signal that precedes it — is a more tractable handle than anything we had before.
Sam: That's the honest read. The percolation framing is a meaningful step toward a rigorous theory of optimization dynamics. The magnification factor, the Reeb graph representation, the link to Discrete Scale Invariance — these are concrete theoretical contributions. The question of how far they generalize is the work that follows.
Alex: Thanks for walking through the mechanism, Sam. This one's worth reading carefully if you work on training dynamics or interpretability — the topological framing opens up some genuinely different questions about what's happening inside the optimization loop.
Sam: Thanks for listening to ResearchPod.