Functional Equivalence and Geometric Diversity in Neural Network Approximations: An Empirical Characterization

This paper investigates the phenomenon of functional non-uniqueness in neural networks—the observation that multiple distinct network architectures can produce nearly identical approximations of the same target function. The authors seek to understand whether these functionally equivalent networks share similar geometric properties in their parameter space or if they represent fundamentally different ways of learning the same underlying function.

The researchers analyze the geometry of the parameter space through the lens of "sloppiness," a concept derived from systems biology that characterizes how model predictions respond to changes in parameters. They utilize the eigenvalue spectrum of the Hessian of the cost function and the effective rank to quantify the dimensionality of the parameter space. The study evaluates a wide range of single-layer and multilayer perceptron (MLP) architectures across various elementary mathematical functions, both in noise-free and noisy conditions, to identify equivalence classes of models that achieve similar approximation accuracy.

The study demonstrates that neural network approximations are inherently over-parameterized. Even within a set of functionally equivalent models, there is significant geometric diversity, meaning that functional similarity does not imply geometric similarity. Crucially, the effective rank of the Hessian remains consistently low across these models, indicating that only a small fraction of the parameter space is actually relevant to the learned representation. This low-dimensional structure persists even when noise is introduced to the data. Based on these findings, the authors propose a new model selection index that balances parameter count, effective rank, and computational cost to identify the most parsimonious and efficient model within an equivalence class.