BACKGROUND: Recent studies have shown that domain-adapted large language models (LLMs) do not consistently outperform general-purpose counterparts on standard medical benchmarks, raising questions about the need for specialized clinical adaptation. METHODS: We systematically compare general and clinical LLMs on a diverse set of multiple choice clinical question answering tasks in English and Spanish. We introduce a perturbation based evaluation benchmark that probes model robustness, instruction following, and sensitivity to adversarial variations. Our evaluation includes, one-step and two-step question transformations, multi prompt testing and instruction guided assessment. We analyze a range of state-of-the-art clinical models and their general-purpose counterparts, focusing on Llama 3.1-based models. Additionally, we introduce Marmoka, a family of lightweight 8B-parameter clinical LLMs for English and Spanish, developed via continual domain-adaptive pretraining on medical corpora and instructions. RESULTS: The experiments show that clinical LLMs do not consistently outperform their general purpose counterparts on English clinical tasks, even under the proposed perturbation based benchmark. However, for the Spanish subsets the proposed Marmoka models obtain better results compared to Llama. CONCLUSIONS: Our results show that, under current short-form MCQA benchmarks, clinical LLMs offer only marginal and unstable improvements over general-purpose models in English, suggesting that existing evaluation frameworks may be insufficient to capture genuine medical expertise. We further find that both general and clinical models exhibit substantial limitations in instruction following and strict output formatting. Finally, we demonstrate that robust medical LLMs can be successfully developed for low-resource languages such as Spanish, as evidenced by the Marmoka models.
Alex: Welcome to another episode of ResearchPod. Sam, you've been looking at some work on AI in medicine—what's this paper about?
Sam: It's called "To Adapt or not to Adapt, Rethinking the Value of Medical Knowledge-Aware Large Language Models" by Ane Domingo-Aldama and colleagues from the University of the Basque Country. The central question is whether we need to spend resources specially training AI language systems for medicine.
Alex: So is the puzzle here that recent studies suggest adapted medical AIs don't reliably beat general ones on standard tests—or are benchmarks missing something?
Sam: Exactly. Large language models are computer systems trained on vast amounts of text to understand and generate responses, much like a very advanced version of the predictive text on your phone that can hold full conversations. When researchers adapt them for medicine, they give these systems extra training on medical books and articles to sharpen their skills for doctor-like tasks. The paper finds that on English medical multiple-choice questions, these clinical versions show only small, inconsistent gains over general models.
Alex: And they checked Spanish too, since it has less training data?
Sam: Yes. For Spanish medical questions, their new lightweight models called Marmoka do outperform general ones. To test deeper, they created tweaks to the questions—like shuffling answer choices or rephrasing them—to check if the AIs truly understand or just memorize patterns. This reveals hidden weaknesses in both types of models.
Alex: So do standard tests fool us into thinking general models are good enough for real doctor work?
Sam: The study suggests yes, at least for English. Benchmarks often pick the best AI answers using a scoring trick called perplexity, which hides flaws in following instructions or handling changes. Their tweaks—like shuffling answer order so the AI can't rely on memorized positions, or swapping labels from A-B-C-D to random letters—show both general and clinical models struggle with strict formats, like outputting just "A" without extra words. For low-resource Spanish, though, Marmoka's adaptation—extra medical text training on smaller 8-billion-parameter models—makes a meaningful difference.
Alex: So perplexity lets models cheat by not really generating the answer themselves?
Sam: Yes. This open-ended matching shows general models drop notably under stresses in English, suggesting they memorize formats more than reason deeply. In Spanish, Marmoka's hybrid training—first soaking up medical texts through continued pretraining, then blending with instruction practice—handles the shakes about twice as well as general models. The paper suggests this proves adaptation's value for low-data languages, without the forgetting issues of heavier tuning.
Alex: That makes sense for a Spanish doctor—the paraphrased question wouldn't trip it up. So how did they train Marmoka to handle those perturbations without falling apart?
Sam: They start with a model already good at following instructions, like Llama 3.1-8B-Instruct. They give it more practice reading huge piles of medical texts—26 billion words in English from sources like PubMed articles and medical wikis, and about 1 billion in Spanish from similar but scarcer collections. This extra reading session, called domain adaptive pretraining, helps the model pick up medical facts and language patterns without starting from scratch. To prevent forgetting how to chat about everyday topics, they mix in general conversation examples. For Spanish Marmoka, they added medical question-practice data too, then merged versions into one stronger model, like mixing recipes for the best cake.
Alex: Like giving a student extra homework on biology but reminding them of English class so they don't blank on grammar.
Sam: Yes. This hybrid keeps the model's helpfulness while boosting medical reliability. In English, gaps stayed small—even with tougher two-step tweaks like summarizing or paraphrasing, clinical models like Marmoka-en only edged ahead modestly. Spanish tells a different story: Marmoka-es handled stresses about twice as well as general ones.
Alex: So for low-resource languages, that lightweight blending avoids heavy downsides like losing broad skills.
Sam: Precisely. It shows adaptation's value shifts by language: marginal for data-rich English, essential for Spanish clinical tasks. The perturbations confirm it's not just memorization—real reasoning edges emerge where data matters most.
Alex: But the paper flags some limits—what do they say?
Sam: A key one is sticking to short multiple-choice quizzes, which test picking answers but miss deeper doctor skills like piecing together patient histories or weighing risks over many steps. Scores come from automatic checks, without doctors reviewing if the reasoning holds up in real cases. They also used just smaller 8-billion-parameter models from one family, so bigger or different designs might shift things. Plus, these are research tools only—not cleared for clinics, as web-sourced data can carry biases from uneven real-world coverage.
Alex: Well said. That's a clear takeaway from this work on rethinking medical language models—questioning big adaptations for English but pointing to targeted ones for underserved areas like Spanish medical care. Thanks, Sam—appreciate the deep dive.
Sam: My pleasure, Alex. Thanks for listening to ResearchPod.