Temporally Consistent Label Interpolation for Robust Surgical Multi-Task Learning under Challenging Conditions

Surgical scene understanding requires modeling both high-level temporal workflows (e.g., phase and step recognition) and fine-grained spatial tasks (e.g., instrument segmentation). However, these tasks suffer from an annotation granularity mismatch: temporal tasks are densely labeled, while spatial tasks are only sparsely annotated due to the high cost of expert labeling. This paper investigates how to bridge this gap to enable effective joint multi-task learning.

The authors propose Flow-guided Annotation for Robust Operating Scenes (FAROS). FAROS addresses the limitations of using foundation models like SAM2 for mask propagation in surgical videos, where appearance-based tracking often fails due to smoke, occlusion, and motion blur. FAROS combines SAM2 with optical flow estimation (RAFT) to detect propagation failures. When an anomaly is detected, the system uses the nearest ground-truth keyframe and optical flow to warp the mask and generate corrective prompts, re-initializing the segmentation model. These densified pseudo-labels are then integrated into a unified Transformer-based multi-task framework that jointly optimizes for phase recognition, step recognition, anticipation, instrument segmentation, and action recognition.

Findings

FAROS significantly improves the quality of pseudo-label generation compared to standalone foundation models, particularly in challenging surgical conditions like instrument exit or blood-induced occlusion. By providing dense spatial supervision, the framework enables better cross-task representation learning. Experiments on the GraSP, MISAW, and AutoLaparo benchmarks demonstrate that this joint optimization leads to superior performance across all spatio-temporal surgical tasks compared to training them independently or with sparse labels alone.

This research provides a scalable solution to the data scarcity problem in surgical AI. By automating the densification of sparse annotations, the method reduces the burden on clinical experts while simultaneously improving the performance of holistic surgical scene understanding systems, which are critical for developing safe and efficient robot-assisted surgical platforms.