ResearchPod Summary
In offline reinforcement learning, researchers often face datasets where rewards are missing not at random (MNAR). This means the probability of a reward being recorded depends on the value of the reward itself (e.g., patients with poor health outcomes skipping follow-up surveys). Standard OPE methods like Importance Sampling or Fitted Q-Evaluation (FQE) fail in these scenarios because they assume the data is missing at random, leading to significant selection bias. This paper addresses how to accurately evaluate a target policy's value when the logged data systematically underreports specific reward outcomes.
To solve this, the authors formalize the MNAR mechanism using a reward-dependent propensity model. They leverage future states as 'shadow variables'—proxies that provide information about the unobserved rewards. By introducing a bridge function, they convert the problem of recovering the conditional mean reward into a conditional moment problem. This allows them to estimate the true reward without needing to explicitly model the complex missingness mechanism, which typically causes high variance in inverse propensity weighting methods. They implement this via a min-max optimization procedure to avoid the double-sampling issues common in reinforcement learning, and then integrate these recovered rewards into an FQE-style estimator.
This work provides a robust framework for high-stakes domains like healthcare and marketing, where data is rarely 'clean' and missingness is often correlated with the outcomes of interest. By avoiding the need for explicit propensity modeling and mitigating the variance issues of traditional methods, this approach enables more reliable policy evaluation in real-world environments where data collection is irregular or censored.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.