UniMedSeg: Unified In-Context Learning for Multi-Paradigm 2D/3D Medical Image Segmentation

Medical image segmentation models are often fragmented, with separate architectures for different prompt types (e.g., interactive points vs. text instructions) and spatial dimensions (2D vs. 3D). This fragmentation prevents models from learning from heterogeneous data sources and limits their ability to generalize across clinical scenarios. The authors ask: Can a single, unified Transformer-based architecture ingest diverse medical inputs—visual examples, geometric prompts, and language instructions—to perform segmentation across both 2D and 3D modalities without task-specific fine-tuning?

UniMedSeg treats all inputs as a unified token sequence. It uses a Transformer-centric backbone that processes target images, visual examples, interactive masks, and language tokens on an equal footing. To handle the high computational cost of long sequences, the authors introduce Decoupled Split Attention, which reduces the attention complexity from quadratic to linear. This mechanism allows the model to maintain focused interactions between specific context-target pairs while still enabling global reasoning. The framework also employs learnable Type Embeddings to distinguish between different input types and uses Rotary Position Embeddings (RoPE) to maintain spatial alignment across 2D and 3D inputs.

UniMedSeg achieves state-of-the-art performance across multiple segmentation paradigms, including visual in-context learning, interactive segmentation, and language-guided segmentation. By unifying these paradigms into a single interface, the model demonstrates strong zero-shot generalization on held-out tasks, including unseen anatomical structures and cross-species targets. The Decoupled Split Attention mechanism successfully enables the model to scale to long-context inputs (such as 3D volumes) while remaining computationally efficient and hardware-friendly.