A Unified Algorithmic Framework for Hybrid Reinforcement Learning in Tabular MDPs with Shifted Transition Dynamics

Hybrid reinforcement learning (RL) aims to improve sample efficiency by combining online interaction with a target environment and offline data from a source environment. A significant challenge arises when the source and target environments have different transition dynamics, rendering standard integration methods ineffective. This paper addresses the question: how can we develop a unified algorithmic framework that provably exploits offline data under general transition shifts?

The authors propose two algorithms: MIN-UCB-VI for regret minimization and MAX-LCB-VI for best policy identification. Both algorithms utilize a model-based approach that incorporates fine-grained bias information—specifically, an upper bound on the L1 distance between transition kernels—to adjust the confidence intervals used during value iteration. This allows the agent to intelligently weight offline data based on its reliability relative to the online environment.

The study provides both instance-dependent and instance-independent theoretical guarantees for the proposed framework. The authors demonstrate that their algorithms achieve optimal regret and sub-optimality bounds, matching established lower bounds. A key contribution is the introduction of a new concentrability metric that jointly characterizes environment bias and policy mismatch, providing a more accurate measure of policy quality under shifted dynamics. The results show that the cumulative bias over time steps significantly impacts online learning performance, and the proposed framework effectively mitigates this by balancing online exploration with biased offline experience.

This work bridges a critical gap in hybrid RL by moving beyond the restrictive assumption that offline and online environments are identical. By providing a robust, theoretically grounded approach to handling transition shifts, this framework offers a more realistic path for deploying RL agents in scenarios where historical data is available but potentially outdated or collected from imperfect simulators. The unified nature of the framework—addressing both regret minimization and best policy identification—makes it a versatile tool for various practical RL applications.