MamaBench: Benchmarking LLM Robustness in Maternal and Child Health Diagnosis through Counterfactual Clinical Perturbation

How robust are large language models (LLMs) when faced with clinically similar patient narratives that require different diagnostic conclusions? Current medical benchmarks evaluate questions in isolation, failing to test whether a model can distinguish between subtle, discriminative clinical parameters. This study investigates the "Einstellung effect"—or diagnostic fixation—where a model correctly diagnoses a base case but fails when a minor, critical detail is altered.

The authors introduce MamaBench, a new benchmark consisting of 434 expert-authored clinical narratives (217 counterfactual pairs) across 371 pathologies in maternal and pediatric health. To address the identified fragility, they propose Evidence-Anchored RAG (EA-RAG), a three-stage inference-time pipeline. Unlike standard RAG, which relies on aggregate similarity, EA-RAG uses: (1) structured clinical parameter extraction, (2) a two-pass retrieval process with coverage auditing to identify and fill information gaps via contrastive sub-queries, and (3) a taxonomy-grounded reasoning scaffold that forces the model to prioritize clinical reasoning over retrieved context.

• Systemic Fragility: Across all tested frontier LLMs, base accuracy significantly overstates robust accuracy by 16-28 percentage points. Standard RAG provides no benefit for counterfactual robustness due to "retrieval stagnation," where the model retrieves nearly identical context for both base and counterfactual cases.
• EA-RAG Performance: The EA-RAG framework reduces the Bias Trap Rate (BTR) by 5.5 percentage points on the strongest model (Claude Sonnet 4.6) without degrading base accuracy.
• Persistent Challenges: Despite these improvements, even the best-performing system fails on approximately 20% of counterfactual pairs, indicating that counterfactual robustness remains an open, critical challenge for clinical AI.