HPR-SAM: Hierarchical Probabilistic Representation Learning for Prompt-free SAM-based Medical Image Segmentation

Prompt-free adaptation of the Segment Anything Model (SAM) for medical imaging often struggles because it relies on deterministic anatomical representations. These representations fail to capture the complex, uncertain nature of anatomical structures. This paper asks: can we improve segmentation accuracy by learning more expressive, probabilistic anatomical representations instead of focusing solely on prompt generation strategies?

The authors introduce the Hierarchical Probabilistic Representation (HPR) framework. Rather than treating anatomical targets as fixed points, HPR models them through three complementary probabilistic modules:

• Distributional Anatomical Representation (DAR): Models global anatomical priors as Gaussian distributions to capture stable semantic characteristics across patients.
• Multi-component Anatomical Representation (MAR): Uses a mixture of probabilistic components to account for intra-structure diversity and morphological variations.
• Local Reliability Representation (LRR): Estimates the reliability of local image tokens by comparing them against class-wise reliability distributions, effectively suppressing noise from ambiguous or low-contrast regions.

These representations are projected into SAM-compatible prompts and decoded into three distinct prediction maps. A final Hierarchical Prediction Fusion (HPF) module adaptively integrates these maps to produce the final segmentation.

Experiments on the Synapse, LA, and PROMISE12 datasets show that HPR-SAM achieves state-of-the-art performance on Synapse and superior results in few-shot settings on LA and PROMISE12. By explicitly modeling anatomical uncertainty and diversity, the framework provides richer semantic cues to the SAM decoder than existing prototype-based methods, leading to more robust and accurate segmentation in challenging medical scenarios.