Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

Training neural networks is typically viewed as a non-convex, discrete optimization problem solved via stochastic gradient descent. This paper proposes a paradigm shift by reformulating the training of shallow neural networks as a continuum variational problem. By working in weighted Sobolev spaces, the authors transform the discrete search for weights into a well-posed problem of finding an optimal parameter density. This approach provides a principled framework to analyze the implicit bias of neural networks without relying on linearization or the lazy-training assumptions common in the Neural Tangent Kernel (NTK) literature.

The authors identify a family of functionals that are globally lambda-convex, ensuring the existence and uniqueness of a stable minimizer. A key theoretical contribution is the proof that this continuum formulation is an exact relaxation of the discrete training problem, meaning there is no Lavrentiev gap—the infimum of the risk is identical whether one optimizes over atomic measures or smooth Sobolev densities. Furthermore, the optimal parameter density exhibits surprising near-C3 regularity, even when the activation function is only locally Lipschitz. This smoothness suggests that neural networks are implicitly attracted to low-dimensional, well-behaved structures in parameter space, providing a concrete mechanism for non-overfitting.

Because the regularized objective is quadratic, the optimal parameter density can be found by solving a single linear system, completely eliminating the need for iterative gradient descent. The authors demonstrate that this approach is not only computationally efficient but also robust to noise and outliers. Finite-width networks of size N are shown to converge to this continuum optimum at an O(1/N) rate, confirming that the continuum model is a faithful representation of practical, discrete neural networks. Empirical results on benchmark datasets show that this variational approach achieves competitive performance with standard deep learning baselines while offering superior stability and interpretability.