Sanjay Basu, Bhairavi Muralidharan, Parth Sheth, Dan Wanek, John W. Morgan, Sadiq Y. Patel
5 min
Care management teams for Medicaid beneficiaries often struggle to prioritize interventions for patients with complex, overlapping medical and social needs. Current practices rely heavily on individual experience and standardized protocols that may not account for the sequential, long-term impact of decisions. This study investigated whether a reinforcement learning model could provide better 'next best step' recommendations than the status quo, effectively optimizing care sequences to reduce acute care events.
The researchers utilized a state-action-reward-state-action (SARSA) reinforcement learning model, which is designed to learn optimal policies in sequential decision-making environments. Unlike off-policy methods, SARSA is an on-policy algorithm, making it safer for clinical settings by ensuring recommendations remain closer to established practice patterns. The model was trained on longitudinal data from 3,175 Medicaid beneficiaries across two states. The researchers performed counterfactual causal inference to compare the SARSA model's performance against the status quo, evaluated fairness across demographic subgroups using equalized odds, and conducted a qualitative grounded theory analysis of cases where the model's recommendations diverged from human judgment.
The SARSA-guided approach significantly outperformed standard care management, achieving an absolute risk reduction of 12 percentage points in acute care events. This resulted in a number needed to treat (NNT) of 8.3, meaning approximately 8 patients would need to receive model-guided care to prevent one acute event. The model was particularly effective for high-risk patients (NNT of 5.2). Furthermore, the model demonstrated improved fairness, reducing disparities in acute event rates across gender and racial/ethnic groups. Qualitative reviews revealed that the model excelled at identifying complex medical-social interactions—such as the link between poor housing quality and respiratory issues—that were frequently overlooked by human care managers.
As Medicaid programs increasingly rely on multidisciplinary teams to address social determinants of health, there is a critical need for decision support tools that can handle complex, longitudinal patient trajectories. This study demonstrates that reinforcement learning can effectively augment human decision-making, leading to better clinical outcomes and more equitable care delivery without deviating into unsafe or impractical recommendations.
BACKGROUND: Multidisciplinary care management teams must rapidly prioritize interventions for patients with complex medical and social needs. Current approaches rely on individual training, judgment, and experience, missing opportunities to learn from longitudinal trajectories and prevent adverse outcomes through recommender systems. OBJECTIVE: This study aims to evaluate whether a reinforcement learning approach could outperform standard care management practices in recommending optimal interventions for patients with complex needs. METHODS: Using data from 3175 Medicaid beneficiaries in care management programs across 2 states from 2023 to 2024, we compared alternative approaches for recommending "next best step" interventions: the standard experience-based approach (status quo) and a state-action-reward-state-action (SARSA) reinforcement learning model. We evaluated performance using clinical impact metrics, conducted counterfactual causal inference analyses to estimate reductions in acute care events, assessed fairness across demographic subgroups, and performed qualitative chart reviews where the models differed. RESULTS: In counterfactual analyses, SARSA-guided care management reduced acute care events by 12 percentage points (95% CI 2.2-21.8 percentage points, a 20.7% relative reduction; P=.02) compared to the status quo approach, with a number needed to treat of 8.3 (95% CI 4.6-45.2) to prevent 1 acute event. The approach showed improved fairness across demographic groups, including gender (3.8% vs 5.3% disparity in acute event rates, reduction 1.5%, 95% CI 0.3%-2.7%) and race and ethnicity (5.6% vs 8.9% disparity, reduction 3.3%, 95% CI 1.1%-5.5%). In qualitative reviews, the SARSA model detected and recommended interventions for specific medical-social interactions, such as respiratory issues associated with poor housing quality or food insecurity in individuals with diabetes. CONCLUSIONS: SARSA-guided care management shows potential to reduce acute care use compared to standard practice. The approach demonstrates how reinforcement learning can improve complex decision-making in situations where patients face concurrent clinical and social factors while maintaining safety and fairness across demographic subgroups.
Alex: Did that systemic view extend to equity? Models that optimize for average outcomes often do it at the expense of marginalized subgroups.
Sam: The authors tracked this explicitly using equalized odds across gender and race. And the result was a genuine win-win: the model not only maintained performance across subgroups but actually reduced outcome disparities. The likely mechanism is that incorporating social determinants gave the model a more accurate picture of which interventions were actually feasible for patients with fewer resources—rather than defaulting to the same clinical playbook regardless of context.
Alex: So where does the causal inference framework become a liability?
Sam: That's the central methodological constraint. The counterfactual framework is susceptible to time-varying unmeasured confounders—factors that influence both which intervention gets chosen and what the outcome is, but aren't captured in those forty-seven features. If something like that is operating, the effect size estimate could be biased in ways that are hard to detect post hoc. The authors acknowledge it, but it's the finding a careful referee would press hardest on.
Alex: And then there's the generalizability question. This was implemented by a single vendor across two states.
Sam: Which is the second major constraint. Medicaid programs vary enormously in what resources are actually available. An intervention sequence that works in a state with robust housing support infrastructure might be undeliverable somewhere with limited community resources. The model essentially learned the local reality of those two states—their action space, their resource availability, their patient population. You can't lift and shift it. Adoption elsewhere would require retraining on local data and redefining the action space to match what's actually available in that community.
Alex: You mentioned a third issue the authors flag—gaming?
Sam: Yes. If care teams learn that the model responds favorably to certain documented inputs, there's an incentive to manipulate the record to get preferred recommendations. The authors suggest randomized auditing as a safeguard, but that's an implementation challenge that sits entirely outside the model itself.
Alex: What would a meaningful next iteration look like?
Sam: The most obvious extension is integrating real-time community resource availability directly into the state vector—live food bank capacity, shelter bed counts, that kind of thing. Right now the model works from relatively static features. If it could see dynamic resource constraints, the sequencing could become genuinely hyper-localized. That's a meaningful step toward closing the gap between knowing a patient is high-risk and actually knowing what to do about it.
Alex: Thanks for listening to ResearchPod.