Unlocking In-Context Learning in Audio-Language Models from Decentralized Medical Audio

Clinical audio diagnosis is typically limited by the need for large, centrally aggregated, and expert-annotated datasets. This paper addresses the challenge of performing open-vocabulary clinical diagnosis in privacy-constrained, low-resource settings where such labels are unavailable. The authors ask: can a multimodal language model learn to perform in-context diagnostic reasoning using only unlabeled audio data distributed across multiple institutions?

The authors propose Federated Self-Contextualization (FSC), a three-stage framework that decouples the acquisition of reasoning skills from the availability of clinical labels. First, the model aligns audio embeddings with a medical language model (MedGemma) using caption-based pretraining. Second, it uses unsupervised K-means clustering on local audio data to create "pseudo-label" episodes—assigning meaningless identifiers like "Mountain Breeze" to audio clusters. This forces the model to learn the abstract skill of mapping acoustic patterns to labels without memorizing specific disease associations. Finally, the model is fine-tuned using LoRA adapters under a federated learning protocol, ensuring that raw patient data never leaves its source institution. At inference, the model uses its pretrained medical knowledge to bridge the gap between the abstract pseudo-labels learned during training and real-world clinical terms.

FSC significantly outperforms existing audio-language baselines in few-shot diagnostic tasks. In a 2-way 2-shot evaluation, FSC achieved 71.6% accuracy, exceeding the strongest centralized baseline by over 9 percentage points. The model demonstrated consistent performance across diverse respiratory and cardiac conditions, suggesting that the learned in-context reasoning skill is generalizable. Ablation studies confirmed that the progressive training pipeline and the use of semantically void pseudo-labels are critical for preventing the model from overfitting to specific training labels and for enabling effective transfer to real clinical terminology.