Sharp Stability Threshold and Certification for Designing Stable Residual Architectures

Modern deep residual architectures rely heavily on normalization layers (e.g., LayerNorm, RMSNorm) to ensure stable training. This paper proposes a theoretical framework that explains this stability through the lens of classical ODE theory and optimal control. The authors identify a sharp stability threshold: for a residual block with velocity field v(x, t), the growth must satisfy ||v(x, t)|| <= c||x||^q + b, where q <= 1. When q > 1, the system enters a supercritical regime where activations can diverge in finite time, leading to training instability regardless of initialization.

The authors establish the q <= 1 threshold using two independent arguments. First, classical ODE theory demonstrates that for q <= 1, the forward flow of the network is globally defined on the interval [0, T], whereas q > 1 allows for velocity fields that cause the solution to blow up. Second, by framing training as an optimal control problem, the authors show that the training optimum is 'bang-bang'—it naturally pushes the network toward the boundary of the admissible velocity class. If that boundary includes supercritical (q > 1) fields, the optimization process will inevitably select them, causing the model to crash.

To make this principle practical, the authors develop an arithmetic of input-magnitude exponents. By cataloging the exponents of standard architectural primitives (like linear layers, attention, and activations) and defining how these exponents transform under operations like addition and composition, researchers can certify the stability of a block by simply calculating its cumulative q-exponent. This allows for the design of stable architectures without relying on trial-and-error placement of normalization layers.