Adam Converges in Nonsmooth Nonconvex Optimization

Adam is a dominant optimizer in machine learning, yet its theoretical convergence properties in nonsmooth, nonconvex settings—which better represent neural network training—have remained elusive. Previous theoretical studies often modified the algorithm by removing the bias-correction step or adding extra operations like clipping. This paper seeks to bridge the gap between theory and practice by providing a convergence guarantee for the classical, unmodified Adam algorithm.

Using the Online-to-Nonconvex Conversion (O2NC) framework, the author analyzes the original Adam algorithm with its bias-correction terms intact. The study establishes a finite-time convergence rate for nonsmooth, nonconvex objectives. Crucially, the analysis accounts for the modern 'heavy-tailed' noise regime, where stochastic gradients have finite $p$-th moments for $p \in (4/3, 2]$, a more realistic assumption than the standard finite-variance condition.

• The author proves that with a randomly scaled learning rate, the original Adam algorithm converges at a rate of $1/T^{2/13}$ for nonsmooth nonconvex optimization.
• In the heavy-tailed noise regime, the convergence rate is generalized to $1/T^{2(3p-4)/(19p-12)}$.
• The theory supports the empirical observation that setting the hyperparameters $\beta_1 = \beta_2$ leads to improved performance.
• The results provide the first theoretical justification for the original Adam algorithm's success in nonsmooth settings without requiring algorithmic modifications like gradient clipping.