Adapt Only When It Pays: Budgeted Decision-Loss Priority for Delayed Online Time-Series Adaptation

In many real-world time-series applications, predictions are made in real-time, but labels (feedback) arrive with significant delays. Furthermore, updating a model consumes limited computational resources. This paper asks: given a hard budget for model updates and delayed feedback, how should a system decide which specific events are worth the cost of an update to minimize downstream decision loss?

To address this, the author introduces ADOWIP (Adapt Only When It Pays), a framework that treats model adaptation as a resource-allocation problem. Instead of updating on every available label or using complex counterfactual probes, ADOWIP uses an observed decision-loss priority gate. This scheduler only triggers an update if the current downstream decision loss exceeds a calibrated empirical quantile and if the system has remaining compute budget. The framework includes a safety variant that can penalize updates based on prediction error to mitigate overfitting risks. The author provides theoretical guarantees for budget feasibility and regret bounds for a projected linear version of the adapter.

ADOWIP was evaluated across several public proxy tasks, including ETT capacity planning, UCI Bike sharing demand, and Capital Bikeshare station rebalancing. The results show that the decision-loss priority gate consistently outperforms baseline strategies—such as always updating, fixed-period updates, and drift-triggered updates—when compute budgets are matched. The framework demonstrated strong performance in capacity-planning and rebalancing tasks, where it achieved significant reductions in decision loss. However, the author notes that performance on secondary tasks like threshold-based alarms and load-index suites was mixed, and experiments involving finance-related diagnostics remained negative, helping to define the current scope of the method.