Vanilla SGD with Momentum Survives Heavy-Tailed Noise: Convergence Analysis without Gradient Clipping or Normalization

Modern machine learning often involves stochastic noise that violates the standard bounded variance assumption, exhibiting "heavy tails" instead. While researchers typically use gradient clipping or normalization to ensure convergence in these settings, this paper investigates whether vanilla SGD with momentum—the industry standard—can converge without these modifications.

The authors analyze the convergence of vanilla SGD with momentum across strongly convex, convex, and nonconvex objective functions. Instead of the traditional L-smoothness assumption, they utilize a Hölder continuous gradient framework. By leveraging specific technical tools for heavy-tailed analysis—such as generalized descent lemmas and moment inequalities—they derive convergence rates in expectation without assuming bounded gradients or employing gradient control mechanisms.

The paper establishes that vanilla SGD with momentum is theoretically capable of converging under heavy-tailed noise. Specifically, for nonconvex objectives, the authors prove a convergence rate of O(T^{-(p-1)/2p}) when the tail index p is known. The analysis confirms that the condition ν + 1 ≤ p is critical for stable convergence, suggesting that Hölder smoothness is a more appropriate framework than standard L-smoothness for heavy-tailed settings. While these rates are slower than those achieved by clipped or normalized SGD, they provide a fundamental baseline for understanding the inherent limitations of vanilla optimization methods.

This work bridges a gap in optimization theory by proving that the most widely used momentum-based optimizers do not strictly require clipping or normalization to converge, even when noise variance is unbounded. By providing a rigorous theoretical foundation for vanilla SGD with momentum, the paper allows researchers to better evaluate the performance gains provided by more complex, robust optimization algorithms.