Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

This paper investigates the approximation capabilities of ReLU neural networks for analytic functions, which are functions that possess infinite smoothness. While much of the existing literature characterizes approximation rates using a single parameter (like total parameter count), this study employs the (N,L)-characterization, which treats network width (N) and depth (L) as independent, flexible parameters. The authors aim to determine how these two architectural choices influence the approximation of analytic functions, specifically looking for rates that reflect the infinite differentiability of the target class.

To achieve this, the authors utilize multivariate Legendre polynomial approximations as a global approximation strategy. They construct specific ReLU network architectures to approximate these polynomials, power functions, and multivariate multiplication. By analyzing these constructions, they derive upper bounds for the approximation error and compare them against lower bounds to assess optimality.

The study demonstrates that analytic functions can be approximated by ReLU networks at a significantly faster rate than functions with finite smoothness. A key insight is that the convergence rate is not uniform across the L-N plane; instead, it varies based on the scaling relationship between depth and width.

Crucially, the findings reveal that depth plays a more critical role than width in the context of analytic function approximation. The authors provide a theorem (Theorem 1) that quantifies this, showing that the approximation rate follows an O(N^{-CL^τ}) form, where τ is a parameter determined by the ratio of depth to width. This result is further supported by a lower bound (Theorem 2), which confirms that the derived upper bounds are nearly optimal in specific regimes. The authors also apply these findings to nonparametric regression, showing that their ReLU network estimators achieve near-minimax convergence rates for analytic target functions.

This work bridges a gap in neural network approximation theory by moving beyond finite-smoothness function classes. By explicitly showing that depth is more important than width for analytic functions, the paper provides theoretical justification for the architectural choices often seen in deep learning, where depth is frequently prioritized to capture complex, smooth underlying structures. The refined network construction techniques for polynomials and multiplication are also valuable contributions that may assist in future research on network expressivity.