ADAM: A METHOD FOR STOCHASTIC OPTIMIZATION

Adam is a first-order, gradient-based optimization algorithm designed for stochastic objective functions. It is particularly well-suited for large-scale machine learning problems characterized by high-dimensional parameter spaces, noisy gradients, or sparse data. The algorithm derives its name from its core mechanism: Adaptive Moment Estimation.

Adam computes individual adaptive learning rates for each parameter by maintaining estimates of the first moment (the mean) and the second raw moment (the uncentered variance) of the gradients. These estimates are updated using exponential moving averages, controlled by two decay rate hyperparameters. To address the bias introduced by initializing these moving averages at zero, the authors implement a bias-correction technique. This ensures that the moment estimates remain accurate, especially during the initial training steps where the bias would otherwise be most pronounced.

Adam combines the strengths of two popular optimization methods: AdaGrad, which excels with sparse gradients, and RMSProp, which is effective in non-stationary, online settings. The algorithm is computationally efficient, requires minimal memory, and is largely invariant to diagonal rescaling of gradients. Empirically, Adam demonstrates robust performance across a variety of models, including logistic regression, deep neural networks, and convolutional neural networks, often outperforming existing stochastic optimization methods in both convergence speed and final performance.