SAMRI-3D: Adapting SAM2 for 3D MRI Segmentation with Global Volume Tokens

Medical image segmentation is challenging due to non-uniform intensities and low soft-tissue contrast, which often render anatomical boundaries invisible on individual MRI slices. While foundation models like SAM2 offer potential, they are typically trained on natural images and struggle with the volumetric continuity and specific geometry of MRI data. This paper asks how to effectively adapt SAM2 for 3D MRI segmentation while addressing the loss of global context inherent in its slice-by-slice streaming memory.

The authors introduce SAMRI-3D, a framework that adapts SAM2 by freezing the Hiera image encoder and fine-tuning only the lightweight decoder and memory modules. To overcome the limitations of SAM2's local-window memory, the authors propose Global Volume Tokens (GVT). These persistent tokens are trained using a Truncated Signed Distance Field (TSDF) reconstruction objective, which forces the model to encode boundary geometry rather than just binary occupancy. This TSDF decoder is used only during training, meaning the GVT adds zero computational cost at inference time.

SAMRI-3D was evaluated on the largest MRI-only benchmark to date, comprising 10,392 volumes across 34 datasets. The model achieved a mean Dice score of 0.78, significantly outperforming zero-shot SAM2 (0.58) and existing medical adaptations like SAMed-2 (0.69) and Medical-SAM2 (0.49). Notably, the model demonstrated robust generalization, showing no performance drop on 8 held-out datasets. Mechanistic analysis confirmed that the TSDF objective specifically improves segmentation in low-contrast sequences where image boundaries are weakest.

This work provides a scalable, efficient way to adapt powerful video-segmentation foundation models to 3D medical imaging. By using geometry-aware training objectives (TSDF) to inject global context into local memory banks, the authors demonstrate that it is possible to achieve high-accuracy segmentation across diverse anatomical domains and MRI sequences without the need for massive, per-task retraining.