Evaluating covariate balance for long time horizon Markov decision processes

Offline reinforcement learning (RL) is increasingly used to derive optimal treatment recommendations from observational healthcare data. However, these applications often rely on assumptions—such as consistency and exchangeability—that are difficult to verify. This paper investigates whether existing offline RL studies in healthcare meet the rigorous diagnostic checks for confounding and model miss-specification that are standard in the causal inference community, specifically focusing on covariate balance.

The authors formalize the definitions of covariate balance and extend them to time-dependent settings, which are common in Markov decision processes (MDPs). They distinguish between two types of balance: conditional covariate balance and weighted covariate balance. Using data from sepsis management studies, the researchers apply these diagnostics to evaluate the robustness of treatment recommendations. They also introduce a Welsh-style denominator for their standardized difference metrics to better account for unequal variances across treatment groups, arguing that standard equal-variance assumptions are overly optimistic in this context.

The analysis reveals that existing offline RL studies for sepsis treatment fail to satisfy the proposed covariate balance tests. The authors demonstrate that the variance ratios between treatment groups diverge significantly over time, suggesting that the underlying assumptions required for valid causal inference are not being met. Consequently, the study concludes that current offline RL applications in this domain cannot be considered statistically robust. The authors emphasize that these diagnostic failures highlight a critical need for more rigorous methodological standards in future research.

This work serves as a cautionary tale for the application of AI in high-stakes medical decision-making. By demonstrating that popular offline RL pipelines may be built on shaky statistical foundations, the authors provide a framework for researchers to audit their own models. It shifts the focus from merely achieving high performance in off-policy evaluation to ensuring that the causal assumptions underpinning those evaluations are actually supported by the data.