Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI | Yexiao He et al. | ResearchPod

Transcript: Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Alex: Most AI models deployed in medicine never learn from a single case they solve. This work shows a frozen model can still get better on the job, if its lessons live outside the model and must pass a test before they're kept.

Sam: Before the how, tell me what I'm listening to. Whose work is it, and what kind?

Alex: It's a method paper from researchers at the University of Maryland and NVIDIA. It's an arXiv preprint dated 2026, tested on six benchmarks with four different base models.

Sam: Plenty of us use these models through an API and can't touch the weights. Why should that group care?

Alex: Because nothing here needs the weights. It works with closed models as well as open ones. And the authors report the learned expertise can be handed to a different model, which then improves without any further learning. So we'll cover what was missing before, the one idea that makes it work, and how far to trust the numbers.

Sam: Start with what was missing. People have tried memory for language models for a while.

Alex: They have. One family saves reflections or strategies from past attempts and just adds them. The authors name Reflexion, ExpeL and ACE. Nothing checks whether a saved lesson actually helps, and it's stored as text, so visual detail gets lost.

Sam: And the other family?

Alex: Methods like EvoSkill and SkillOpt do check. They propose updates to a skill document and pick the ones that score well on a fixed validation set. The worry is that if you keep selecting against the same validation cases, you end up overfitting them. Fine-tuning is the third option, but that needs weights and training.

Sam: So the gap is learning that's checked, but not checked against the same cases forever.

Alex: That's the heart of it. The framework keeps three things outside the model. First, a Skill, which is a plain-language how-to document for reasoning and tool use. Second, a Knowledge Memory, which stores facts, each backed by evidence. Third, for image tasks, a Multimodal Knowledge Base, meaning a library of earlier images stored with their known answers.

Sam: A how-to document that keeps growing sounds like it would bloat and contradict itself after a few hundred cases.

Alex: The authors make the same point, so the optimizer rewrites the whole document each time instead of appending. Their example is a chest X-ray. One batch suggests ordering an X-ray for cough and fever. A later batch shows it's often unnecessary when vital signs and the lung exam are normal. The rewrite merges both into one rule: order the X-ray for cough and fever when vitals or the lung exam are abnormal.

Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI

Yexiao He, Yucheng Tang, Pengfei Guo, Yufan He, Andriy Myronenko, Can Zhao, Ang Li, Daguang Xu, Dong Yang

Oct 6, 2026·6:57·arxiv·Original Paper

Model-Agnostic LearningMultimodal Medical AIDeployment ExperienceExternal ExpertiseMultimodal Knowledge Base (MMKB)Experience-Driven Learning

Listen

7 min

Summary

Research Question and Approach

Large language models (LLMs) and vision-language models (VLMs) typically remain frozen after deployment, preventing them from learning from the clinical cases they process. This limitation is particularly critical in medicine, where clinical evidence and practice evolve rapidly. The authors propose a framework that allows frozen models to improve their performance during deployment by accumulating external expertise rather than updating model weights. The system maintains three specific components: a Skill for reasoning guidance and tool use, a Knowledge Memory for verified facts, and a Multimodal Knowledge Base (MMKB) for visual references.

Methodology and Expertise Components

Skill: An executable document that guides the model's reasoning steps and tool-use procedures. This document is updated after each batch of cases based on performance analysis.
Knowledge Memory: A storage system for verified medical facts, which are proposed as candidates after processing cases and tested for support by trusted evidence before inclusion.
MMKB: A storage library of previously processed visual cases, including images, source answers, and explanations. The model is prompted to explicitly compare visual cues from these reference cases with the current input to ground its decisions.
Validation Strategy: To prevent overfitting or the accumulation of harmful patterns, every candidate update—whether a new Skill or added Knowledge—is tested against a buffer of past and new cases. Only updates that demonstrably improve outcomes without degrading performance on historical data are permanently integrated.

Main Findings

The framework consistently improved performance across six benchmarks, including interactive clinical diagnosis, medical reasoning, and visual logic tasks. Key findings include:

Performance Gains: The system achieved up to a 34.2% improvement in online deployment performance over the base model across various medical benchmarks.
Generalization and Transfer: Expertise learned by one model could be transferred to other base models without additional training, demonstrating that the external expertise captures generalizable medical strategies rather than model-specific corrections.
Robustness: Unlike methods that update parameters or append unlimited context, the authors' validation-based, rewriting approach prevented performance degradation on past cases and mitigated the risk of overfitting.

Key Terms and Definitions

Model-Agnostic Learning: A method that improves model performance by modifying external context (prompts, knowledge, or tools) rather than updating the underlying model parameters.
Skill: A natural-language procedural document that provides the frozen model with instructions on reasoning steps, tool usage, and decision-making constraints.
Multimodal Knowledge Base (MMKB): A storage component that retains visual evidence, source answers, and explanations from past cases to support grounded comparison in future image-based medical tasks.
Deployment Experience: The stream of cases processed by a model during its operational phase, which provides the training signals and performance feedback used to generate new external expertise.
Validation Buffer: A representative set of historical and new clinical cases used as a quality control mechanism to ensure that any proposed expertise update does not regress performance on past benchmarks.

Abstract

Large language models (LLMs) and vision-language models (VLMs) are usually frozen after deployment, so they do not learn from the cases they solve. This is especially concerning in medicine, where new clinical evidence, updated guidelines, and new therapies can change established practice. Fine-tuning can update the model, but it requires access to model weights and additional training. Parameter-free methods avoid training, but they may overfit a fixed validation set, lack reliable domain knowledge, or lose visual details by saving experience only as text. To address these limitations, we present a model-agnostic framework that allows frozen LLMs and VLMs to learn from deployment experience through three forms of external expertise: a Skill that guides reasoning and tool use, a Knowledge Memory that stores reliable facts supported by earlier cases or trusted external evidence, and a Multimodal Knowledge Base that keeps visual examples and guides the model to relate each retrieved case to the current image. Instead of relying on a fixed validation set, a validation strategy keeps an update only if it helps on new cases without degrading performance on earlier ones. Across six benchmarks covering clinical diagnosis, clinical workflows, medical reasoning, and medical and non-medical visual reasoning, and with four open-weight and closed-source base models, our framework improves performance during online deployment by up to 34.2% over the base model on medical tasks, generalizes to unseen cases, transfers to other models without further optimization, and works in non-medical domains.

Frozen Models, Evolving Expertise: Model-Agnostic Learning from Deployment Experience for Multimodal Medical AI: Research Paper Podcast