Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models

Multiple Instance Learning (MIL) is the standard approach for analyzing whole-slide images (WSIs) in computational pathology. However, these models are typically trained from scratch for each specific task, which often leads to overfitting, unstable optimization, and poor transferability. The authors investigate whether pretraining these lightweight MIL aggregators using knowledge distillation from larger, more powerful pathology foundation models can provide a more robust and efficient starting point for downstream clinical applications.

The authors introduce a multi-teacher distillation framework that uses two state-of-the-art slide-level foundation models, TITAN and CARE, as teachers. Because these teachers have different representation spaces and concentration levels, the authors propose an Angular Dispersion Normalized (ADN) distillation loss. This loss normalizes the alignment error by the running angular dispersion of each teacher's distribution, preventing any single teacher from dominating the optimization process. The framework was used to pretrain nine different MIL architectures, ranging from attention-based and Transformer-based models to graph-based and state-space models, using the TCGA-UT-8K dataset.

The distilled MIL aggregators were evaluated across 15 diverse clinical tasks, including cancer subtyping, biomarker prediction, and mutation detection. The results demonstrate that pretraining consistently outperforms from-scratch training across linear-probing, full-parameter fine-tuning, and few-shot learning scenarios. Notably, the pretrained MIL models often outperformed the teacher models themselves, suggesting that the distillation process effectively captures and adapts the teachers' knowledge into the more compact, task-efficient MIL architectures.

This work provides a practical pathway for deploying lightweight MIL models in clinical settings where annotated data are scarce. By decoupling the learning of general-purpose slide representations from task-specific fine-tuning, the authors enable practitioners to leverage the power of large-scale foundation models without the computational burden of fine-tuning massive, high-capacity encoders for every individual task.