Black-Box Assisted Regression: Phase Transitions and Minimax Optimality

Modern machine learning often involves using a fixed, pretrained foundation model as a black-box predictor for a downstream task. Because the internal parameters and training data of these models are often inaccessible, the learner must decide how to use these predictions safely. This paper asks: how can we optimally combine a fixed black-box predictor with limited labeled data to minimize risk, while ensuring we do not perform worse than the black-box itself (avoiding negative transfer)?

The author models this as a nonparametric regression problem where the target function is the sum of a fixed black-box predictor and an unknown residual. The learner has access to a small labeled dataset and must decide whether to trust the black-box or to learn a correction. The proposed Safe Residual Estimator follows a two-step process: it learns a residual correction on a training set and then uses a separate validation set to perform a binary selection—either keeping the original black-box prediction or adding the learned correction. This binary choice is specifically designed to avoid the instability of continuous mixing weights and to provide a formal guarantee against negative transfer.

The paper establishes a finite-sample minimax characterization of the problem. It proves that the risk is governed by a phase transition: when the black-box is highly accurate (prior-dominated regime), the risk is dominated by the prior error; when the black-box is biased (sample-dominated regime), the risk is governed by the standard nonparametric rate for learning the residual. The Safe Residual Estimator is shown to be minimax-optimal, matching the leading risk term up to an additive validation-selection cost. Experiments on synthetic data and real-world tasks (CIFAR-100 with CLIP and AG News with Qwen3-8B) confirm that this residual-correction strategy effectively adapts to the quality of the black-box prior.