Convergence of Gradient Descent for General Neural Network Architectures Beyond the NTK Regime

Research Question and Approach

Understanding the training dynamics of deep neural networks is challenging because modern architectures, such as transformers and residual networks, are highly nonlinear and do not satisfy the global smoothness or convexity assumptions required by classical optimization theory. While the Neural Tangent Kernel (NTK) framework provides theoretical guarantees, it relies on infinite-width limits and infinitesimal learning rates, which fail to capture the feature-learning behavior observed in practice. This paper proposes a new convergence framework that analyzes gradient descent (GD) at the level of network blocks, allowing for a more realistic treatment of modern, finite-width architectures.

Methodology

The author introduces three key technical pillars to analyze GD trajectories beyond the NTK regime:

1. Polynomial Generalized Smoothness: Instead of global Lipschitz smoothness, the paper uses a block-level regularity condition that is preserved under common operations like addition, multiplication, and composition. This allows the framework to handle complex layers like normalization and attention.
2. Analytic Genericity: By leveraging the fact that many neural network components are real-analytic, the author uses measure-zero arguments to show that the loss landscape behaves well almost everywhere in the parameter space.
3. Local Relaxed Dissipativity: The paper introduces a condition that characterizes the loss landscape near stationary points, ensuring that the gradient descent trajectory does not stagnate.

The paper proves that for almost all initializations, GD with regular learning rates converges to the neighborhood of a stationary point for a broad class of neural networks. A significant insight is that the required learning rate scale is determined by the network's depth and effective bottleneck dimensions, rather than the largest hidden width. Furthermore, the author shows that residual connections and function composition play a structural role in restoring nondegeneracy, which helps explain why these architectures are easier to train. Finally, the paper demonstrates that under this framework, stationary points are generically global minimizers.