Active Offline-to-Online Reinforcement Learning

In offline-to-online reinforcement learning (O2O-RL), researchers often pretrain multiple candidate policies offline and then fine-tune them using limited online interactions. However, because fine-tuning performance is highly sensitive to hyperparameter choices and environmental conditions, committing to a single policy is risky. This paper addresses the challenge of how to optimally allocate a limited online interaction budget to both evaluate candidate policies and improve them through fine-tuning.

The authors formulate the problem as a regret-minimization task where the goal is to identify the best-performing policy after a fixed number of fine-tuning iterations. They propose an active selection strategy that uses an upper-confidence-bound (UCB) criterion. After each online episode, the system evaluates the current policy, fits a local linear model to forecast its future performance, and updates the UCB. The system then dynamically switches to the policy with the highest UCB, effectively balancing the exploration of different candidates with the exploitation of the most promising ones.

The study demonstrates that actively selecting and fine-tuning policies is significantly more effective than either committing to a single policy or distributing the interaction budget equally across all candidates. Experimental results on a suite of simulated robotics benchmarks show that the proposed approach consistently outperforms standard O2O-RL baselines. The authors also provide a theoretical lower bound on regret, illustrating how the number of candidates and the interaction budget influence the difficulty of the selection problem.

This work provides a practical solution for deploying reinforcement learning in real-world systems where online interaction is expensive, hazardous, or time-consuming. By treating policy selection and fine-tuning as a joint, adaptive process, the framework allows for more robust performance in nonstationary environments, moving the field closer to reliable real-world autonomy.