Jing Hao, Weinan Zhou, Zhihong Lu, Yaling Shen, Yiming Xu, Yefeng Liu, Siyuan Yan, Shuaibo Li, Ming Dai, Junjun He, Zuyi Zhu, Jinjie Gu, Lijun Wu, Jian Wang
8 min
Abstract
Reliable dermatology AI requires access to verifiable evidence distributed across heterogeneous textual and visual sources. However, conventional retrieval-augmented generation (RAG) typically relies on one-shot retrieval from flat corpora, limiting its ability to adapt retrieval granularity, explore multi-hop relations, and integrate multimodal clinical evidence. We introduce SkinMind, a multimodal agentic RAG framework that equips policy models with human-inspired graph probing for autonomous evidence seeking. SkinMind organizes retrieval as an iterative interaction between an orchestrator and a graph environment. Through eight hierarchical graph-probing tools, the orchestrator progressively identifies relevant entities, inspects node-level information, traverses relations and paths, and retrieves source-text and visual evidence. This progressive-disclosure design allows the model to adaptively determine the retrieval modality, exploration direction, and evidence granularity while constructing a query-specific evidence subgraph. To support autonomous graph probing, we construct SkinMind-KG from a 62M-token corpus of dermatology guidelines and textbooks and 423K images, guided by a dermatologist-informed schema and a conflict-resolution mechanism. The resulting graph contains over 1.0M nodes and 2.0M edges, unifying textual, relational, and visual evidence for retrieval. Across six dermatology VQA benchmarks and two policy-model settings, SkinMind improves the average score over direct answering by 8.70 and 8.89 points with GPT-5.4-mini and Gemini-2.5-Flash, respectively, while consistently outperforming competing RAG methods. Together, SkinMind and SkinMind-KG provide an effective foundation for evidence-grounded multimodal reasoning in dermatology.
Sam: That separates deciding where to look from reading everything there. What does the underlying resource contain?
Alex: The authors build a graph with about a million nodes. It draws on dermatology guidelines and textbooks, alongside clinical images. Extracted textual claims retain links to supporting passages, so the model can return from a summary to its source context.
Sam: Those connections are themselves machine-generated. How do the authors handle contradictory information?
Alex: Their construction process distinguishes genuine contradictions from claims about different clinical circumstances. For detected conflicts, a model searches for additional evidence and attempts resolution. Ambiguous cases go to experts. That is a documented resolution process, not proof that every graph statement is correct.
Sam: Make the navigation concrete. What happens in an actual diagnostic example?
Alex: In the paper's successful example, the model must identify a photographed skin condition from answer options. It first retrieves similar images and inspects their metadata. One result has a mismatch between its caption and aligned disease label, so that result isn't straightforward support for a diagnosis.
Sam: If a retrieved image has questionable metadata, what other evidence does the system pursue?
Alex: It reads descriptions of lichen amyloidosis and xerotic eczema, competing diagnoses in the question. Then it follows morphology relationships—connections describing lesion shape and appearance. Retrieved features include clustered papules, meaning small raised bumps, and a rough, cobblestone appearance.
Sam: So the useful move isn't just finding a similar photograph. It's checking whether the disease's described features fit the observed lesions.
Alex: That trajectory reaches lichen amyloidosis, the benchmark's correct answer. Direct answering and LightRAG select other options in this example. It's an illustration of the mechanism, though, not the main evidence that the method works.
Sam: Then let's move to the comparison. How do they separate better navigation from simply giving SkinMind better references?
Alex: They test six dermatology visual-question-answering benchmarks, meaning questions involving images. All retrieval methods use the same knowledge resource and shared retrieval backend. Within each answering-model setting, they also share task instructions and generation settings.
Sam: And they don't rely on a single answering model to make the approach look good?
Alex: They test both a GPT mini model and a Gemini Flash model. SkinMind improves the average score over direct answering by roughly nine points in each setting. It also beats the competing retrieval methods across the tested benchmarks.
Sam: Direct answering leaves outside knowledge unused. How much of the advantage survives against a stronger retrieval competitor?
Alex: Against LightRAG, the average gain is about six points with the GPT model. The advantage also holds with Gemini, although it's smaller. That supports the authors' argument that access to knowledge isn't enough; how the system explores it matters.
Sam: Medical image collections can overlap. Did they check whether retrieval was effectively supplying the test photograph?
Alex: They remove exact image duplicates and filter potential near-duplicates using image similarity. Those rules apply across the retrieval methods. Most benchmarks use multiple-choice accuracy; the free-response benchmark uses a model judge comparing answers with references, without seeing the input image.
Sam: That makes the comparison more credible within these tests. But the scoring still measures benchmark answers, not whether clinicians or patients benefit.
Alex: That's the boundary I would keep. The component-removal tests add supporting evidence: disabling graph exploration, image retrieval, or selective disclosure lowers accuracy on the selected diagnostic benchmarks. Those tests support complementary roles for the components, within a narrower evaluation.
Sam: Selective disclosure sounds economical. Does it actually reduce the computational bill?
Alex: It reduces retrieved evidence, not total token use. On the free-response benchmark with Gemini, SkinMind averages about fourteen hundred evidence tokens per question. LightRAG uses about seventy-nine hundred. Tokens are the units of text processed by the model.
Sam: But repeated decisions also require inputs and outputs. Does the smaller evidence bundle outweigh that extra interaction?
Alex: No: SkinMind uses roughly twice as many total tokens as LightRAG in that analysis. Earlier interactions and tool instructions contribute to repeated inputs. Its efficiency claim is about selecting compact evidence, not making the entire process cheaper.
Sam: There's another risk: a model could navigate very carefully around its favourite diagnosis. Do the failures show that happening?
Alex: They do. In one failure, exploration centres on cellular neurothekeoma, a candidate skin tumour. The model checks features supporting that candidate but never explores pilomatricoma, the correct option. In another, it favours keratoacanthoma over basal cell carcinoma while leaving growth history and central keratin unverified.
Sam: Those aren't simply missing-reference problems. The model has evidence, but doesn't complete the comparison that would make its decision defensible.
Alex: That's consistent with the authors' analysis. They suggest improving tool-use strategies to encourage more targeted, comparative exploration. Giving a model navigation tools does not ensure that it uses them to challenge its preferred answer.
Sam: Who should read the full document, and where should they start?
Alex: Researchers building medical retrieval systems or interactive graph search should read it. Start with Hierarchical Graph-Probing Interfaces to see what choices the model actually gets. Then read Main Results and Context Efficiency Analysis together, so compact evidence isn't mistaken for lower overall cost.
Sam: And the section that keeps the reliability claim in perspective?
Alex: The appendix's Failure Case Analysis. It shows the distance between collecting relevant facts and distinguishing diagnoses. For everyone else, remember this: better retrieval helps, but reliability still depends on asking what would distinguish the alternatives.
Sam: Thanks for listening.