Mental health disorders represent a burgeoning global public health challenge. While Large Language Models (LLMs) have demonstrated potential in psychiatric assessment, their clinical utility is severely constrained by benchmarks that lack ecological validity and fine-grained diagnostic supervision. To bridge this gap, we introduce \textbf{MentalDx Bench}, the first benchmark dedicated to disorder-level psychiatric diagnosis within real-world clinical settings. Comprising 712 de-identified electronic health records annotated by board-certified psychiatrists under ICD-11 guidelines, the benchmark covers 76 disorders across 16 diagnostic categories. Evaluation of 18 LLMs reveals a critical \textit{paradigm misalignment}: strong performance at coarse diagnostic categorization contrasts with systematic failure at disorder-level diagnosis, underscoring a gap between pattern-based modeling and clinical hypothetico-deductive reasoning. In response, we propose \textbf{MentalSeek-Dx}, a medical-specialized LLM trained to internalize this clinical reasoning process through supervised trajectory construction and curriculum-based reinforcement learning. Experiments on MentalDx Bench demonstrate that MentalSeek-Dx achieves state-of-the-art (SOTA) performance with only 14B parameters, establishing a clinically grounded framework for reliable psychiatric diagnosis.
Alex: Welcome to another episode of ResearchPod. Today, Sam, we're diving into some work on AI and mental health diagnosis—what's this particular study about?
Sam: This paper, called MentalSeek-Dx, tackles a big challenge in using AI for psychiatry. It introduces the first real-world test set for spotting specific mental health disorders from doctors' notes, and shows how current AI models fall short on the details even if they're good at broad guesses. The key puzzle is that these AIs nail big categories like mood disorders but crash when pinpointing exact conditions like schizophrenia.
Alex: So the core problem here is that AI handles rough mental health labels okay, but struggles with the precise diagnoses doctors need? Like, it might say "mood issue" but miss the specific disorder?
Sam: Exactly. Mental health affects over a billion people worldwide, but there's a huge shortage of psychiatrists, leading to delayed care. Current large language models do well on coarse categories, but on fine-grained disorders using real electronic health records, they drop sharply. This gap comes from AIs relying on simple keyword matching instead of step-by-step doctor-like thinking.
Alex: Right, so even medical-tuned AIs aren't bridging that? And they built a new benchmark to prove it?
Sam: Yes, they created MentalDx Bench with 712 de-identified patient records from clinics, labeled by board-certified psychiatrists following ICD-11 guidelines. ICD-11 is the World Health Organization's standard list of diseases, including 76 psychiatric disorders across 16 categories like anxiety or schizophrenia. Testing 18 models showed even giants like Qwen3-Max excel at categories but falter on disorders. Expert review of errors pinned it on poor reasoning, like ignoring symptoms or weak differential diagnosis, where doctors rule out similar conditions.
Alex: Huh—that's a clear drop-off. So the study argues this reveals a deeper mismatch in how AIs think versus clinicians?
Sam: Yes, large language models lean on quick pattern associations from training data, like spotting keywords, rather than the step-by-step logic clinicians use. To fix this, the researchers built MentalSeek-Dx in two main stages: first, creating a special training set of 15,000 real patient cases, processed through a three-step pipeline using another AI called DeepSeek-R1 and a knowledge base of ICD-11 disorder details. Step one picks the best matching broad category for symptoms; step two lists likely disorders with their criteria; step three rewrites the case to align closely with the true category and disorder, all done securely on-site at the clinic.
Alex: Okay, so they refine raw patient notes into cleaner examples that match doctor thinking? But how do they turn those into actual training for the model?
Sam: They use those refined cases to generate full reasoning paths, breaking diagnosis into three parts: first, sorting symptoms into categories by matching them to known patterns; second, listing possible disorders as hypotheses based on those symptoms and category features; third, comparing hypotheses against full criteria to pick the best one, like ruling out look-alikes. Only complete, correct paths become the supervised fine-tuning data. This teaches the model to follow clinician logic, not just guess.
Alex: That sounds structured—like building a habit of thinking in stages. Does adding rewards during training make it even sharper?
Sam: Precisely. After initial training on those paths, they apply reinforcement learning—a method where the model tries options, gets scores for good ones, and adjusts to favor them more. Here, it's process-based with rewards at each stage: one for right category, one for ranking the true disorder high in hypotheses, and one for the final correct pick. They use group relative policy optimization, comparing batches of tries and normalizing rewards so the model learns relative quality.
Alex: So rewards push precision at every step, not just the end?
Sam: Yes. To build skills gradually, like school subjects from easy to hard, they use curriculum learning: over five training rounds, early ones weight category rewards highest, later ones boost hypothesis and final diagnosis weights. This stabilizes learning.
Alex: Huh. And the results?
Sam: The 14B-parameter MentalSeek-Dx hits state-of-the-art, topping models over eight times larger. Ablations confirm each part adds gains—reasoning paths alone boost scores substantially over plain training, and this RL refines them further, especially on tough cases. Experts rated its logic higher too, for rigor and evidence use.
Alex: Meaningful progress with smaller models. But any catches?
Sam: The paper notes limits: it uses only text from records, missing in-person cues like tone or gestures in real exams. It's strictly a support tool for doctors, not a replacement, due to risks like rare disorder biases or AI errors.
Alex: Those limits make sense—they keep expectations realistic. But to get the reasoning paths right in the first place, they must have drawn on some detailed guide for what counts as a disorder, right?
Sam: Correct. The researchers built a structured knowledge base—think of it as a clear reference book organized like a family tree, with broad groups at the top, like mood disorders, branching into specific ones below, such as bipolar type I. For each entry, it lists everyday descriptions of key signs, like persistent low mood or hallucinations, plus how they affect daily life, and strict checklists from ICD-11 to match symptoms exactly and rule out close matches. Experts helped craft it from official guidelines, so it's reliable for guiding the AI's step-by-step checks.
Alex: Like a detective's notebook with suspect profiles and alibis. Does pulling from that base during training prevent the model from mixing up similar conditions?
Sam: Yes—it feeds directly into generating those trajectories. When sorting symptoms or ranking hypotheses, the system retrieves matching details, like how anxiety involves excessive worry about real situations, unlike schizophrenia's reality breaks. This grounds the process in clinical facts, cutting errors from vague guesses. The paper suggests this alignment helps the smaller model outperform giants by focusing on logic over raw size.
Alex: So it's not just more data, but smarter structure that bridges the gap to real clinics. How did the trained model actually stack up against those bigger ones on the test cases?
Sam: The 14-billion-parameter version outperformed giants over eight times its size on hundreds of real patient records. The paper suggests it's because the training enforces step-by-step logic over sheer scale.
Alex: What were the main ways other models went wrong, according to the experts?
Sam: Experts reviewed hundreds of mistakes and found most stemmed from three issues: spotting symptoms right but drawing the wrong conclusion from them; listing possible conditions but missing the real one; skipping key details from the notes altogether. These added up to over two-thirds of errors. It shows models often guess loosely instead of checking facts systematically.
Alex: Like detectives jumping to suspects without alibis. Did the new training fix those specific flaws?
Sam: Yes, and further tweaks in the reward system helped. They compared giving points just for the end diagnosis versus rewarding each step. The step-by-step version, with weights shifting to harder parts over time, gave smoother progress.
Alex: And experts checked the quality of that reasoning too?
Sam: They did—blind ratings from a dozen doctors on things like logical flow matching clinic steps, solid evidence use from guidelines, covering all clues without gaps, and ruling out rivals properly. MentalSeek-Dx scored higher across these than baselines, confirming its paths feel more like real psychiatrist work.
Alex: So pulling it all together, this work shows that teaching AI to follow a doctor's step-by-step logic—catalog symptoms, list suspects, check alibis—lets a smaller model spot precise disorders better than huge ones relying on quick pattern guesses.
Sam: Yes, the evidence from expert reviews confirms it. A case study highlights this—one big model jumps from hallucinations straight to schizophrenia, skipping alternatives, while the trained one methodically narrows options using criteria, avoiding that trap. In places with doctor shortages, this could flag subtle issues early from notes alone—a meaningful aid, though always checked by humans. The study underscores reasoning structure over raw power for trustworthy psychiatric aid.
Alex: A solid, measured advance then. Thanks, Sam—that's our look at bridging AI to real psychiatric diagnosis. Thanks for listening to ResearchPod.