Zhiwei Bai, Jiajie Zhao, Zhangchen Zhou, Zhi-Qin John Xu, Yaoyu Zhang
12 min
Abstract
Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the "natural" auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.
Alex: Okay, so v_t decouples and decays on its own, boosting the step like cruise control kicking in harder as resistance drops? But doesn't that ramp-up risk overshooting if it goes too far?
Sam: Precisely—the paper shows discretization adds stability limits. They define effective sharpness as a measure combining step size and local flatness; if it exceeds 2, steps overshoot like braking too late on ice. This leads to a one-dimensional map for its evolution, revealing three regimes: stable fixed point for steady linear convergence, period-doubling oscillations that still converge but wobble, and divergence into chaos. For RMSProp, stability holds if β₂ exceeds a critical value like 0.0625 for k=4, which typical settings do. Theorem 5.7 then proves global linear convergence under those conditions, with rate tied to β₂ raised to a power depending on k.
Alex: And those regimes match the full Adam phase diagram?
Sam: Yes—theory and experiments align closely, as low final loss zones overlap stable regions, confirmed by high decoupling ratios there. In practice, this explains Adam's edge on mixed landscapes with flat spots, like Transformers, without manual schedules. The upshot is adaptivity turns a curse into a predictable strength.
Alex: So that non-trivial fixed point you mentioned earlier—where does it come from in the dynamics, and how does the paper confirm it's stable for typical settings?
Sam: To track the behavior near the minimum, the authors define two key ratios from Adam's internal variables. The first is how much the momentum m_t exceeds the current gradient g_t—they label this ω_t. The second compares the local flatness to the square root of the second moment v_t, essentially gauging if the adaptive step size matches the landscape's sharpness; this is λ_t. At the non-trivial fixed point, these settle to specific values where ω* λ* stays constant below 2/η, keeping steps from overshooting, and x shrinks steadily.
Alex: Okay, so λ_t acts like a check on whether the step is tuned to the flatness. But what pins down if that point is stable or leads to wobbles?
Sam: Theorem 4.1 lays it out: the point exists and is meaningful if β₁ is less than β₂ raised to k over 2 times (k-2) squared. Stability comes from the Jacobian matrix there having spectral radius under 1—all its growth factors inside the unit circle, like eigenvalues of a system that pull back to equilibrium rather than spiral out. This holds in a window for β₁, and when stable, the ratio x_{t+1}/x_t approaches β₁ to the power 2(k-2)^2, giving linear convergence. The paper simplifies the full matrix but proves these bounds exactly.
Alex: And those phase diagrams—do they show this window matching real runs for k=4 and 6?
Sam: Yes—the theoretical diagram predicts stable zones where loss hits machine precision, and the empirical one from simulations overlays almost perfectly for those k values. It visualizes how β₁ below about β₂ to the power k over 2(k-2)^2 keeps things in the good regime, explaining Adam's reliability without tweaks. This ties back to why it outperforms on Transformer landscapes: adaptivity implicitly ramps effective learning rates just right on flats.
Alex: Walk me through λ_t itself—what exactly does it measure in plain terms, and why does its settling matter for the steady progress?
Sam: Think of the landscape's flatness near the bottom as how sharply it curves—steeper means bigger nudges from gradients, flatter means tinier ones. The optimizer's step size has to match that: too small and you crawl, too big and you bounce past. λ_t puts those together by taking the local flatness measure and dividing by the square root of v_t—the running average of past gradient squares that scales the step. Researchers call this the effective curvature. When it settles below 2 over the learning rate η, steps stay safe and shrink x steadily.
Alex: Got it—like a speed limit tuned to road grip. So the proofs show it converges to a specific value under those β₂ conditions?
Sam: Yes—Lemma B.2 proves that if β₂ is above a threshold but below 1, and starting λ_0 under 1/η, then λ_t heads to exactly (1-β₁)/(2(k-2)^2 η). They show this by viewing its update as mostly a map whose fixed point λ* is stable since its slope sits between -1 and 1 there. A follow-up lemma confirms global pull to that point from any safe start. Theorem 5.7 seals it: x_t ratios approach β₁^{2(k-2)^2}, locking in linear shrinkage.
Alex: Huh—and the diagrams capture those stable zones hitting machine precision loss?
Sam: Precisely—the theoretical plot predicts low-loss regions where β₁ stays under β₂^{k/(2(k-2)^2)}, and experiments overlay tight for k=4,6, with high decoupling there. This matches why adaptive methods like Adam beat GD or momentum's sublinear crawl—those stick to power-law decay. Adam implicitly ramps effective rates exponentially through v_t decay. The paper suggests this explains Adam's strength on Transformer flats without hand-tuned ramps.
Alex: That automatic boost makes sense for those highly flat polynomials like k=4 or 6. But what about simpler bowl-shaped losses, like a plain quadratic—does the same logic hold there, or does something break?
Sam: The paper contrasts that with quadratics, where L(x) is just half x squared—a smooth bowl with constant curve everywhere. Here, Adam lacks a stable non-trivial fixed point for linear convergence like in higher k; instead, it has an unstable trivial one at zero. Experiments show RMSProp speeds superlinearly at first as v_t decouples and decays freely, mimicking an exponential step-size ramp, but then spikes from overshooting.
Alex: So momentum in full Adam caps that super-speed to steady linear progress? Without letting it runaway?
Sam: Exactly—Lemma F.2 proves momentum acts as a bottleneck: even with growing steps, m_t's memory slows x_t to exponential decay bounded by (1-β₁ minus growth rate over 2) times t. Experiments confirm: v_t decouples from g_t squared, fading at β₂'s rate, but m_t stays tightly coupled to g_t throughout, tracking the shrinking slope without dominating. This keeps updates balanced, hitting linear rates matching theory.
Alex: Those quadratic phase diagrams—do they show low loss where coupling holds?
Sam: Yes—heatmaps of min and final loss drop in high-β₂ zones with moderate β₁, where v_t/g_t^2 ratios stay low, signaling decoupling without excess. Bigger β₁ spikes loss earlier from overshoot; low β₂ lags v_t, blocking deep convergence. For k=4,6 zoom-ins reveal some "unstable" spots still converge via bounded period-doubling—sawtooth wobbles under the sharpness threshold—explaining Adam's real-world reliability.
Alex: So even quadratics highlight the balance: decoupling boosts speed, coupling prevents chaos. Ties back to why Adam handles deep nets steadily.
Alex: So pulling it all together, this mechanism gives Adam a real edge on those flat landscapes common in deep nets.
Sam: Yes—the phase diagrams show theoretical stable zones matching experiments precisely for k=4 and 6, where low final losses hit machine precision and decoupling ratios stay high. This frames why adaptive optimizers like Adam reach linear rates, turning sublinear slogs into reliable shrinkage.
Alex: And those real architecture tests—do they back this up beyond polynomials?
Sam: Experiments compare tuned SGD and Adam across models: comparable on ReLU feedforward nets and CNNs, but Adam pulls ahead on Softmax nets and Transformers—about twice the loss drop in steps on the latter. Hessian spectra there reveal highly degenerate tails, flat like the polynomials. Raw curves confirm Adam's robustness over SGD's sensitivity to learning rate picks. It suggests the v_t-g_t squared decoupling amplifies effective rates exponentially just where needed.
Alex: Right—so practical win on Transformer training. But any catches in the analysis?
Sam: The work focuses on one-dimensional, deterministic polynomials, so stochastic noise from real data batches, high-dimensional interactions across parameters, and non-polynomial losses like exponentials in Transformers remain open. These could couple directions differently or add variance that shifts phases. Still, the core logic holds promise for designing optimizers via these phase maps—auto-tuning betas to architecture flats, ditching manual ramps.
Alex: A solid step toward understanding why Adam works so steadily in practice, grounded in those dynamics. Thanks for listening to ResearchPod.