Learning from the Unseen: Offline Reinforcement Learning with Hidden Actions

In many real-world applications, such as healthcare or complex administrative systems, the actions recorded in offline datasets are often noisy proxies rather than the true actions taken. Standard offline reinforcement learning (RL) algorithms assume observed actions are error-free; when this assumption is violated, these methods produce biased and unreliable policy evaluations. This paper addresses the fundamental challenge of performing off-policy evaluation (OPE) when the true actions are hidden.

The authors propose LURE (Learning from the Unseen: Robust Estimator), a novel framework for OPE with latent actions. The core insight is that the next state in a Markov decision process acts as a natural proxy for the hidden action, as it is generated based on the true action taken. By exploiting this relationship, the authors establish the identifiability of the policy value. They derive an influence-function-based estimator that is multiply robust, meaning it remains consistent if various nuisance components—such as the measurement model, reward function, or density ratio—are correctly specified. To implement this, they introduce an expectation-maximization (EM) style algorithm that iteratively estimates the latent action posterior and the required nuisance functions.

LURE provides a statistically valid way to evaluate policies in the presence of hidden actions. The authors prove that the estimator is asymptotically normal, which allows for the construction of valid confidence intervals and hypothesis tests. Through simulations and an application to sepsis management using the MIMIC-III database, the authors demonstrate that LURE significantly outperforms naive methods that ignore action misclassification, providing more accurate and robust estimates of policy performance.

This work is the first to formally address the hidden-action problem in offline RL. It bridges the gap between standard RL and the statistical literature on measurement error, providing a principled approach for researchers working with imperfectly recorded data. By enabling reliable policy evaluation in settings like electronic health records, LURE helps ensure that data-driven decision-making remains robust to the noise inherent in real-world data collection.