ResearchPod Summary
Large language models (LLMs) and vision-language models (VLMs) typically remain frozen after deployment, preventing them from learning from the clinical cases they process. This limitation is particularly critical in medicine, where clinical evidence and practice evolve rapidly. The authors propose a framework that allows frozen models to improve their performance during deployment by accumulating external expertise rather than updating model weights. The system maintains three specific components: a Skill for reasoning guidance and tool use, a Knowledge Memory for verified facts, and a Multimodal Knowledge Base (MMKB) for visual references.
The framework consistently improved performance across six benchmarks, including interactive clinical diagnosis, medical reasoning, and visual logic tasks. Key findings include:
Alex: Most AI models deployed in medicine never learn from a single case they solve. This work shows a frozen model can still get better on the job, if its lessons live outside the model and must pass a test before they're kept.
Sam: Before the how, tell me what I'm listening to. Whose work is it, and what kind?
Alex: It's a method paper from researchers at the University of Maryland and NVIDIA. It's an arXiv preprint dated 2026, tested on six benchmarks with four different base models.
Sam: Plenty of us use these models through an API and can't touch the weights. Why should that group care?
Alex: Because nothing here needs the weights. It works with closed models as well as open ones. And the authors report the learned expertise can be handed to a different model, which then improves without any further learning. So we'll cover what was missing before, the one idea that makes it work, and how far to trust the numbers.
Sam: Start with what was missing. People have tried memory for language models for a while.
Alex: They have. One family saves reflections or strategies from past attempts and just adds them. The authors name Reflexion, ExpeL and ACE. Nothing checks whether a saved lesson actually helps, and it's stored as text, so visual detail gets lost.
Sam: And the other family?
Alex: Methods like EvoSkill and SkillOpt do check. They propose updates to a skill document and pick the ones that score well on a fixed validation set. The worry is that if you keep selecting against the same validation cases, you end up overfitting them. Fine-tuning is the third option, but that needs weights and training.
Sam: So the gap is learning that's checked, but not checked against the same cases forever.
Alex: That's the heart of it. The framework keeps three things outside the model. First, a Skill, which is a plain-language how-to document for reasoning and tool use. Second, a Knowledge Memory, which stores facts, each backed by evidence. Third, for image tasks, a Multimodal Knowledge Base, meaning a library of earlier images stored with their known answers.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: A how-to document that keeps growing sounds like it would bloat and contradict itself after a few hundred cases.
Alex: The authors make the same point, so the optimizer rewrites the whole document each time instead of appending. Their example is a chest X-ray. One batch suggests ordering an X-ray for cough and fever. A later batch shows it's often unnecessary when vital signs and the lung exam are normal. The rewrite merges both into one rule: order the X-ray for cough and fever when vitals or the lung exam are abnormal.
Sam: Who does the rewriting, and when?
Alex: Cases arrive in batches of thirty-two. The model answers a whole batch first, then the dataset's reference answers and scores are released. Then a separate optimizer model, GPT-5.6-Sol, reads the scored trajectories and proposes a new Skill and a few new facts.
Sam: Facts drawn from a handful of cases worry me in medicine. What stops one lucky answer becoming a rule?
Alex: A verification step. A proposed fact must quote its supporting cases, and normally needs more than one. A single case is enough only with direct support from a trusted outside source, such as PubMed. Patient-specific observations and individual answer keys are excluded.
Sam: Now the gate itself. How does it avoid the fixed-validation-set trap?
Alex: Every candidate update is compared against the current version on two sets. One is the next batch of new cases. The other is a buffer of about thirty past cases, refreshed over time so it covers the kinds of questions seen so far. The update is kept only if it improves the new batch on average, doesn't lower the past buffer on average, and wins at least as many cases as it loses on both. Since the new batch is always different, it's never the same validation set twice.
Sam: And the image library. Isn't that just retrieval-augmented generation with pictures?
Alex: The retrieval is fairly standard. What's different is the instruction. For each retrieved reference, the model has to explain how that image supports its known answer. Then it states which observations carry over to the current image, which differ, and what's unknown. Notably, completed cases go in with the dataset's answer, not the model's prediction, so even a case the model got wrong becomes a useful reference.
Sam: Let's get to results. What's the headline comparison?
Alex: During online deployment, the framework beat the plain base model in all twenty-four benchmark and model pairs. Its average gain over the base model was about eighteen percent, against about six percent for ACE. On the medical tasks, the best gain was about thirty-four percent, on an interactive diagnosis benchmark called AgentClinic.
Sam: Online gains could just be the system memorizing the stream. Did they test on cases it never learned from?
Alex: Yes. They learned on roughly the first seventy percent of the stream, froze everything, and tested on the rest. The average gain held at about twenty-one percent. SkillOpt managed about eight, and ACE about two. SkillOpt and ACE also fell below the base model more often; the framework did so only once.
Sam: Is the gate really doing the work, or is it the three stores?
Alex: They ran the ablation. With validation switched off, so every update is accepted, scores dropped on every benchmark and ended up below the plain base model on all six. Removing the Skill hurt most on multi-step clinical workflows. Removing the image library wiped out nearly all the gain on visual tasks.
Sam: And the comparison against ordinary image retrieval?
Alex: Same model, same retrieved references. Just handing them over as context ranged from slightly worse to slightly better than the base model, and hurt on two tasks. The compare-each-reference instruction improved all four image tasks, by up to about twenty-four percent.
Sam: What about moving the expertise to another model?
Alex: Transferred unchanged, it improved the target model in thirty-three of thirty-six cases. Expertise learned with the open-weight Qwen model, for instance, lifted the weaker GPT-4o-mini by about forty-seven percent on AgentClinic.
Sam: So where would you hold back?
Alex: Three places. Gains were small, under about five percent, on the benchmarks of independent medical questions, which the authors think lean more on the model's own knowledge. The splits are custom splits of public data, not official hidden test sets. And robustness to case order was checked on only two benchmarks with one model. I'd add that the learning depends on released reference answers and a strong optimizer model, which real clinics won't always have.
Sam: So who should read the full paper, and where should they start?
Alex: Anyone building deployed clinical agents or self-improving language-model systems. Start with the section on validating updates during deployment, then the ablation, especially the run without validation. If you work with images, read the comparison of the reference library against standard retrieval.
Sam: And for everyone else?
Alex: The model can stay frozen. What it learns just has to pass a test on new cases and old ones before it's kept.
Sam: A useful rule for anyone letting systems learn on the job.