Ane G. Domingo-Aldama, Iker De La Iglesia, Maitane Urruela, Aitziber Atutxa, Ander Barrena
5 min
Abstract
BACKGROUND: Recent studies have shown that domain-adapted large language models (LLMs) do not consistently outperform general-purpose counterparts on standard medical benchmarks, raising questions about the need for specialized clinical adaptation. METHODS: We systematically compare general and clinical LLMs on a diverse set of multiple choice clinical question answering tasks in English and Spanish. We introduce a perturbation based evaluation benchmark that probes model robustness, instruction following, and sensitivity to adversarial variations. Our evaluation includes, one-step and two-step question transformations, multi prompt testing and instruction guided assessment. We analyze a range of state-of-the-art clinical models and their general-purpose counterparts, focusing on Llama 3.1-based models. Additionally, we introduce Marmoka, a family of lightweight 8B-parameter clinical LLMs for English and Spanish, developed via continual domain-adaptive pretraining on medical corpora and instructions. RESULTS: The experiments show that clinical LLMs do not consistently outperform their general purpose counterparts on English clinical tasks, even under the proposed perturbation based benchmark. However, for the Spanish subsets the proposed Marmoka models obtain better results compared to Llama. CONCLUSIONS: Our results show that, under current short-form MCQA benchmarks, clinical LLMs offer only marginal and unstable improvements over general-purpose models in English, suggesting that existing evaluation frameworks may be insufficient to capture genuine medical expertise. We further find that both general and clinical models exhibit substantial limitations in instruction following and strict output formatting. Finally, we demonstrate that robust medical LLMs can be successfully developed for low-resource languages such as Spanish, as evidenced by the Marmoka models.
Alex: That makes sense for a Spanish doctor—the paraphrased question wouldn't trip it up. So how did they train Marmoka to handle those perturbations without falling apart?
Sam: They start with a model already good at following instructions, like Llama 3.1-8B-Instruct. They give it more practice reading huge piles of medical texts—26 billion words in English from sources like PubMed articles and medical wikis, and about 1 billion in Spanish from similar but scarcer collections. This extra reading session, called domain adaptive pretraining, helps the model pick up medical facts and language patterns without starting from scratch. To prevent forgetting how to chat about everyday topics, they mix in general conversation examples. For Spanish Marmoka, they added medical question-practice data too, then merged versions into one stronger model, like mixing recipes for the best cake.
Alex: Like giving a student extra homework on biology but reminding them of English class so they don't blank on grammar.
Sam: Yes. This hybrid keeps the model's helpfulness while boosting medical reliability. In English, gaps stayed small—even with tougher two-step tweaks like summarizing or paraphrasing, clinical models like Marmoka-en only edged ahead modestly. Spanish tells a different story: Marmoka-es handled stresses about twice as well as general ones.
Alex: So for low-resource languages, that lightweight blending avoids heavy downsides like losing broad skills.
Sam: Precisely. It shows adaptation's value shifts by language: marginal for data-rich English, essential for Spanish clinical tasks. The perturbations confirm it's not just memorization—real reasoning edges emerge where data matters most.
Alex: But the paper flags some limits—what do they say?
Sam: A key one is sticking to short multiple-choice quizzes, which test picking answers but miss deeper doctor skills like piecing together patient histories or weighing risks over many steps. Scores come from automatic checks, without doctors reviewing if the reasoning holds up in real cases. They also used just smaller 8-billion-parameter models from one family, so bigger or different designs might shift things. Plus, these are research tools only—not cleared for clinics, as web-sourced data can carry biases from uneven real-world coverage.
Alex: Well said. That's a clear takeaway from this work on rethinking medical language models—questioning big adaptations for English but pointing to targeted ones for underserved areas like Spanish medical care. Thanks, Sam—appreciate the deep dive.
Sam: My pleasure, Alex. Thanks for listening to ResearchPod.