Evaluating Large Language Models on Misconceptions in Multi-Turn Medical Conversations

Patients often approach medical information sources with preconceived notions or misconceptions about their health. In a clinical setting, a physician's role is not merely to answer a question but to identify and correct the underlying false belief. As LLMs become common tools for health information, it is critical to determine if they can maintain this standard of care. Unlike static, single-turn benchmarks, real-world medical conversations are iterative and multi-turn, meaning misconceptions can emerge, persist, or evolve as the patient and the model interact.

To investigate this, the authors introduce ThReadMed-QA, a new dataset comprising 2,437 conversation threads and 8,204 question-answer pairs sourced from the r/AskDocs subreddit. This dataset is unique because it captures authentic, patient-authored queries and follow-ups alongside verified physician responses. By preserving the natural flow of these interactions, the dataset allows researchers to evaluate how well models handle misconceptions as they arise throughout a dialogue, rather than treating each question in isolation.

The researchers evaluated five state-of-the-art LLMs—including GPT-5, GPT-4o, and Claude Haiku—using a rubric-based LLM-as-a-Judge framework. The results reveal a consistent and concerning pattern: even models that perform well on the first turn show a sharp decline in their ability to correct misconceptions as the conversation continues. For example, some models saw their correction rates drop from approximately 85% on initial questions to roughly 50% within just two follow-up turns. An oracle analysis confirmed that this degradation is largely due to error propagation, where the model's own previous, less-than-perfect responses negatively influence its subsequent reasoning.