Knowledge Cascade: Reverse Knowledge Distillation on Nonparametric Multivariate Functional Estimation

As machine learning models and datasets grow in size and complexity, the computational cost of training becomes a significant bottleneck. While traditional knowledge distillation (KD) compresses a trained teacher model into a smaller student model, it does not solve the problem of the initial, expensive training of the teacher. This paper asks whether the distillation process can be reversed: can a small, computationally efficient student model provide the necessary guidance to train a larger, more complex teacher model?

The authors introduce Knowledge Cascade (KCas), a framework that treats the student model as a pilot study. In the context of nonparametric multivariate functional estimation (specifically smoothing splines), the primary computational challenge is selecting multiple smoothing parameters. KCas selects these parameters on a small subsample of the data and then uses asymptotic scaling laws to extrapolate the optimal parameters for the full-sample teacher model. This allows the teacher to be fitted without the prohibitive cost of full-sample hyperparameter tuning.

KCas significantly reduces the computational complexity of smoothing-parameter selection, for instance, from O(n^3) to O(n^(3/4)) in specific smoothing spline applications, while maintaining statistical accuracy. The authors demonstrate that this approach is not limited to smoothing splines; it is also effective for kernel density estimation and deep learning hyperparameter transfer. Experiments show that KCas not only saves substantial computational resources but can occasionally outperform standard full-sample tuning by avoiding the instability associated with overly adaptive parameter selection on large datasets.

KCas provides a theoretically grounded paradigm for efficient model development in data-rich, compute-constrained environments. By shifting the focus from model compression to efficient model construction, it offers a scalable alternative for researchers working with high-dimensional data or complex functional models where traditional cross-validation is computationally infeasible.