Adaptive deep nonparametric regression from dependent data under covariate shift

In many real-world applications, such as healthcare or natural language processing, the distribution of training data (source) differs from the distribution of testing data (target), a phenomenon known as covariate shift. While deep neural networks (DNNs) have shown promise in regression, most existing theoretical guarantees assume independent and identically distributed (i.i.d.) data and identical source-target distributions. This paper addresses the challenge of performing nonparametric quantile and Huber regression under covariate shift using dependent data.

The authors propose a sparse-penalized deep neural network (SPDNN) estimator. To handle the discrepancy between source and target distributions, they employ a reweighting strategy using the density ratio $r(x) = q(x)/p(x)$. When this ratio is unknown, they implement a two-step pre-training procedure: first, they estimate the density ratio using a least squares SPDNN, and second, they use this estimate to reweight the training data for the final regression function estimation. The framework is designed to be robust to various forms of data dependence, including $\phi$-mixing, strong mixing, and $\mathcal{C}$-mixing processes, by utilizing a generalized Bernstein-type inequality.

The study establishes non-asymptotic error bounds for the proposed SPDNN estimators within the class of Hölder smooth functions. The authors demonstrate that these estimators are adaptive and can achieve the minimax optimal convergence rate (up to a logarithmic factor) for both quantile and Huber regression. This optimality holds even when the density ratio is unbounded, provided it has finite moments. By incorporating sparse-penalized regularization, the approach avoids the practical difficulties associated with standard sparsity constraints, allowing the model to adapt to the unknown smoothness of the target function.