We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper that tries to explain one of the more persistent puzzles in deep learning: why training so often feels like nothing is happening, and then suddenly everything changes.
Sam: The paper is "Percolation Dynamics in Optimization," and its central argument is that we've been mischaracterizing SGD. We treat it as a smooth descent through a loss landscape, but the authors make the case that it's actually a non-equilibrium process — one that proceeds through discrete structural events rather than continuous adjustment.
Alex: So this is a mechanistic account of grokking? The phenomenon where a model spends thousands of epochs apparently memorizing, then abruptly generalizes?
Sam: Exactly. And the mechanism they propose is percolation. The key insight is that architectural symmetries — the fact that neurons in the same layer are functionally interchangeable — don't just create redundancy. They create invariant sets in the loss landscape. Regions where the optimizer is effectively trapped, because any permutation of those neurons leaves the loss unchanged.
Alex: So the optimizer isn't descending freely. It's navigating a landscape that has these flat, symmetric plateaus built into it by the architecture itself.
Sam: Right. And what happens is that as training progresses, the gradient pressure forces neurons to break that symmetry collectively. Think of it like a social network where everyone starts acting independently. The constraints of the architecture gradually force them into cliques — and when a clique forms, it forms all at once. Not gradually. The whole subnetwork snaps into alignment simultaneously.
Alex: And that snap is what shows up as a spike in the gradient variance?
Sam: Yes. The authors track this by mapping the training trajectory onto a Reeb graph — a structure that captures how the topology of the connected components in parameter space changes over time. What they find is that these merger events don't happen at arbitrary intervals. They follow a geometric cascade. The authors call this Discrete Scale Invariance: each merge happens at a time that's a fixed multiplicative factor after the previous one.
Alex: So there's a ratio — a magnification factor — that governs the timing of these transitions?
Sam: That's the claim. And they extract this factor empirically from tasks like modular arithmetic, where grokking is well-documented. The relative variance of the gradient diverges in a predictable pattern before each phase transition. Which means, in principle, you can see it coming.
Alex: That's a meaningful shift in how you'd think about training diagnostics. Instead of watching the loss curve and hoping, you'd be monitoring the topology of the parameter space directly.
Sam: Exactly. The practical upshot is a diagnostic signal — a way to forecast the onset of generalization before it happens. You could imagine a learning rate scheduler that doesn't operate on heuristics or fixed epoch counts, but instead responds to that divergence signal. Adjust hyperparameters when the system is approaching a transition, rather than after the fact.
Alex: Okay, but let me push on the foundations here. The percolation framing assumes these merging components are macroscopic — that the phase transition is real and not just a finite-size artifact. How solid is that assumption?
Sam: That's the central open question the authors themselves flag. The thermodynamic limit — what happens to this structure as the number of parameters grows without bound — is unresolved. If the merging components are small relative to the network size, the apparent discontinuity might smooth out at scale. Grokking might not be a sharp transition in large models; it might be a gradual crossover that just looks sharp at the scales we currently train at.
Alex: Which would significantly change how much weight you'd put on the topological interpretation.
Sam: It would. The authors point to empirical evidence from transformers — that the cascade structure holds up as model complexity increases — but that's not a proof. It's a suggestive pattern. The gap between the finite-size observations and a rigorous thermodynamic theory is real, and a careful referee would push hard on it.
Alex: So the load-bearing result is the geometric cascade in the variance signal, and the open question is whether that cascade survives to the scale where it would actually matter for frontier models.
Sam: That's a fair summary. What the paper establishes clearly is the mechanism at tractable scales and the diagnostic signal it produces. What it doesn't establish — and doesn't claim to — is that the same sharp transitions persist in the regime where most practitioners are actually working.
Alex: Still, as a framework it's doing something useful. It moves the conversation away from "add more data, add more compute" and toward understanding the internal structural dynamics of the network. Even if the thermodynamic limit is fuzzy, having a principled account of why grokking happens — and a signal that precedes it — is a more tractable handle than anything we had before.
Sam: That's the honest read. The percolation framing is a meaningful step toward a rigorous theory of optimization dynamics. The magnification factor, the Reeb graph representation, the link to Discrete Scale Invariance — these are concrete theoretical contributions. The question of how far they generalize is the work that follows.
Alex: Thanks for walking through the mechanism, Sam. This one's worth reading carefully if you work on training dynamics or interpretability — the topological framing opens up some genuinely different questions about what's happening inside the optimization loop.
Sam: Thanks for listening to ResearchPod.