Xiao Sun, Yuming Yang, Junnan Zhu, Jiang Zhong, Xinyu Zhou, Kaiwen Wei
9 min
Abstract
Mental health disorders represent a burgeoning global public health challenge. While Large Language Models (LLMs) have demonstrated potential in psychiatric assessment, their clinical utility is severely constrained by benchmarks that lack ecological validity and fine-grained diagnostic supervision. To bridge this gap, we introduce \textbf{MentalDx Bench}, the first benchmark dedicated to disorder-level psychiatric diagnosis within real-world clinical settings. Comprising 712 de-identified electronic health records annotated by board-certified psychiatrists under ICD-11 guidelines, the benchmark covers 76 disorders across 16 diagnostic categories. Evaluation of 18 LLMs reveals a critical \textit{paradigm misalignment}: strong performance at coarse diagnostic categorization contrasts with systematic failure at disorder-level diagnosis, underscoring a gap between pattern-based modeling and clinical hypothetico-deductive reasoning. In response, we propose \textbf{MentalSeek-Dx}, a medical-specialized LLM trained to internalize this clinical reasoning process through supervised trajectory construction and curriculum-based reinforcement learning. Experiments on MentalDx Bench demonstrate that MentalSeek-Dx achieves state-of-the-art (SOTA) performance with only 14B parameters, establishing a clinically grounded framework for reliable psychiatric diagnosis.
Sam: They use those refined cases to generate full reasoning paths, breaking diagnosis into three parts: first, sorting symptoms into categories by matching them to known patterns; second, listing possible disorders as hypotheses based on those symptoms and category features; third, comparing hypotheses against full criteria to pick the best one, like ruling out look-alikes. Only complete, correct paths become the supervised fine-tuning data. This teaches the model to follow clinician logic, not just guess.
Alex: That sounds structured—like building a habit of thinking in stages. Does adding rewards during training make it even sharper?
Sam: Precisely. After initial training on those paths, they apply reinforcement learning—a method where the model tries options, gets scores for good ones, and adjusts to favor them more. Here, it's process-based with rewards at each stage: one for right category, one for ranking the true disorder high in hypotheses, and one for the final correct pick. They use group relative policy optimization, comparing batches of tries and normalizing rewards so the model learns relative quality.
Alex: So rewards push precision at every step, not just the end?
Sam: Yes. To build skills gradually, like school subjects from easy to hard, they use curriculum learning: over five training rounds, early ones weight category rewards highest, later ones boost hypothesis and final diagnosis weights. This stabilizes learning.
Alex: Huh. And the results?
Sam: The 14B-parameter MentalSeek-Dx hits state-of-the-art, topping models over eight times larger. Ablations confirm each part adds gains—reasoning paths alone boost scores substantially over plain training, and this RL refines them further, especially on tough cases. Experts rated its logic higher too, for rigor and evidence use.
Alex: Meaningful progress with smaller models. But any catches?
Sam: The paper notes limits: it uses only text from records, missing in-person cues like tone or gestures in real exams. It's strictly a support tool for doctors, not a replacement, due to risks like rare disorder biases or AI errors.
Alex: Those limits make sense—they keep expectations realistic. But to get the reasoning paths right in the first place, they must have drawn on some detailed guide for what counts as a disorder, right?
Sam: Correct. The researchers built a structured knowledge base—think of it as a clear reference book organized like a family tree, with broad groups at the top, like mood disorders, branching into specific ones below, such as bipolar type I. For each entry, it lists everyday descriptions of key signs, like persistent low mood or hallucinations, plus how they affect daily life, and strict checklists from ICD-11 to match symptoms exactly and rule out close matches. Experts helped craft it from official guidelines, so it's reliable for guiding the AI's step-by-step checks.
Alex: Like a detective's notebook with suspect profiles and alibis. Does pulling from that base during training prevent the model from mixing up similar conditions?
Sam: Yes—it feeds directly into generating those trajectories. When sorting symptoms or ranking hypotheses, the system retrieves matching details, like how anxiety involves excessive worry about real situations, unlike schizophrenia's reality breaks. This grounds the process in clinical facts, cutting errors from vague guesses. The paper suggests this alignment helps the smaller model outperform giants by focusing on logic over raw size.
Alex: So it's not just more data, but smarter structure that bridges the gap to real clinics. How did the trained model actually stack up against those bigger ones on the test cases?
Sam: The 14-billion-parameter version outperformed giants over eight times its size on hundreds of real patient records. The paper suggests it's because the training enforces step-by-step logic over sheer scale.
Alex: What were the main ways other models went wrong, according to the experts?
Sam: Experts reviewed hundreds of mistakes and found most stemmed from three issues: spotting symptoms right but drawing the wrong conclusion from them; listing possible conditions but missing the real one; skipping key details from the notes altogether. These added up to over two-thirds of errors. It shows models often guess loosely instead of checking facts systematically.
Alex: Like detectives jumping to suspects without alibis. Did the new training fix those specific flaws?
Sam: Yes, and further tweaks in the reward system helped. They compared giving points just for the end diagnosis versus rewarding each step. The step-by-step version, with weights shifting to harder parts over time, gave smoother progress.
Alex: And experts checked the quality of that reasoning too?
Sam: They did—blind ratings from a dozen doctors on things like logical flow matching clinic steps, solid evidence use from guidelines, covering all clues without gaps, and ruling out rivals properly. MentalSeek-Dx scored higher across these than baselines, confirming its paths feel more like real psychiatrist work.
Alex: So pulling it all together, this work shows that teaching AI to follow a doctor's step-by-step logic—catalog symptoms, list suspects, check alibis—lets a smaller model spot precise disorders better than huge ones relying on quick pattern guesses.
Sam: Yes, the evidence from expert reviews confirms it. A case study highlights this—one big model jumps from hallucinations straight to schizophrenia, skipping alternatives, while the trained one methodically narrows options using criteria, avoiding that trap. In places with doctor shortages, this could flag subtle issues early from notes alone—a meaningful aid, though always checked by humans. The study underscores reasoning structure over raw power for trustworthy psychiatric aid.
Alex: A solid, measured advance then. Thanks, Sam—that's our look at bridging AI to real psychiatric diagnosis. Thanks for listening to ResearchPod.