Understanding From Human Perspective: A Multi-agent System for Interactive Egocentric Medical Image Segmentation

Interactive egocentric medical image segmentation (IEMIS) is essential for smart-glasses-assisted medical review, where a clinician's verbal instructions must be translated into precise visual segmentations. However, this task faces two major hurdles: semantic ambiguity, where instructions like "segment the kidney" fail to specify which side, and visual variability, where head motion and viewpoint changes cause segmentation masks to drift or jump across video frames. This paper introduces EgoMed-Agent to address these challenges.

EgoMed-Agent utilizes a multi-agent architecture that decomposes the segmentation process into two distinct workflows:

1. Target Confirmation Workflow: A Detection Agent identifies candidate medical objects, and a Confirmation Agent grounds the user's instruction against these candidates. If the grounding is uncertain, the system proactively asks the user for clarification, ensuring the target is explicitly defined before segmentation begins.

2. Localization-Guided Propagation Workflow: Once the target is confirmed, a Propagation Agent tracks the mask across frames. Simultaneously, the Detection Agent re-localizes the target in each frame. The system performs a consistency evaluation between the propagated mask and the localized target, automatically re-initializing the mask if the two diverge to prevent drift.

By integrating these workflows, EgoMed-Agent provides a robust, training-free paradigm that can be applied across diverse medical imaging modalities. The system's ability to resolve ambiguity through interaction and maintain stability through localization-guided correction makes it a practical solution for real-world clinical settings, where reliable, hands-free visual evidence is critical for decision-making.