Selective Time Series Forecasting via Metalearning

Deep learning models for time series forecasting often exhibit non-uniform accuracy, struggling significantly with certain samples. While reject-option mechanisms exist for classification and regression, they are underutilized in forecasting. This paper addresses the challenge of identifying 'hard-to-predict' periods in advance to allow models to abstain from unreliable forecasts, specifically focusing on the need for a rejection mechanism that generalizes across different time series domains.

The authors introduce a selective forecasting framework that decouples the rejection decision from the forecasting model itself. Instead of relying on model-specific uncertainty proxies like prediction interval width—which are often tied to the training domain—the authors use a metalearning approach. They extract structural features (e.g., trend, seasonality, complexity) from recent lags of a time series and train a metamodel to predict the empirical percentile rank of the forecasting error. Because these features are domain-agnostic and percentile ranks are scale-invariant, the rejection mechanism can be transferred to new, heterogeneous datasets without requiring full retraining.

Experiments demonstrate that the metamodel consistently ranks forecast origins by their expected error, effectively identifying challenging periods. In both in-domain and transfer learning settings, the framework outperforms traditional baselines such as prediction interval width and residual-based heuristics. The authors show that the metamodel achieves performance close to an 'oracle' (which knows the true errors), and that domain adaptation—fine-tuning the metamodel on a small subset of target data—significantly improves performance when transferring across different time series collections.

This work provides a robust, model-agnostic way to improve the reliability of time series forecasting systems. By enabling pre-forecast screening, practitioners can avoid high-risk predictions in production environments, effectively trading off coverage for higher accuracy. The ability to transfer this rejection logic across domains makes it a practical tool for real-world applications where labeled data for training new models is scarce.