Transfer Learning in Nonparametric Regression with Deep ReLU Networks

Nonparametric regression is a flexible framework that models complex data relationships without assuming rigid functional forms. However, standard methods often struggle with high dimensionality—a challenge known as the curse of dimensionality. Transfer learning addresses data scarcity in target domains by leveraging information from related source groups. While transfer learning has been heavily studied in parametric settings, its theoretical properties for deep neural networks in nonparametric regression remain less understood. This paper investigates nonparametric regression with data split into multiple groups that share a common underlying structure.

The authors propose a two-stage offset learning procedure inspired by pretraining and residual modeling. Under the assumption that each group's conditional mean is the sum of a shared overall mean function and a group-specific offset, the estimation proceeds in two distinct steps. In the first stage, data from all groups are pooled together to estimate the overall mean function across the entire population. In the second stage, data from each specific target group are used to estimate the group-specific offset relative to the overall mean. The final group-level estimator is constructed by adding the estimated overall mean and the estimated group offset.

The theoretical analysis establishes general upper bounds on the L2 error for a broad class of nonparametric estimators. When the framework is instantiated using deep ReLU networks under hierarchical composition models, explicit convergence rates are derived. The results demonstrate that deep neural networks preserve their ability to overcome the curse of dimensionality within this transfer learning setup. Furthermore, the analysis identifies specific regimes where transfer learning achieves strictly faster convergence rates than single-group estimation, providing theoretical justification for the empirical success of pretrained models.