GAFT: Geo-Anchored Fine-Tuning for Hazard Identification from Rare Failures

Autonomous robots in unstructured environments often encounter rare, hazardous structures—such as fallen logs or roots—that can cause entrapment or high-centering. Because these failure events are costly to collect and provide only outcome-level labels, standard supervised fine-tuning often leads to shortcut learning, where models rely on incidental visual cues (like lighting or background texture) rather than the physical structures that actually cause failure. This paper asks how to effectively adapt vision foundation models for hazard identification when labeled failure data are extremely scarce.

To address this, the authors propose Geo-Anchored Fine-Tuning (GAFT), a two-stage parameter-efficient adaptation method. In the first stage, GAFT uses an RGB-depth traversal dataset (without hazard labels) to construct a proximity-weighted geometric prior. This prior emphasizes structures protruding above the ground that are close to the robot. During adaptation, the model uses LoRA (Low-Rank Adaptation) to align the vision model's spatial attention-rollout map with this geometric prior, while a KL-divergence anchor preserves the pretrained representations of the DINOv2 backbone. In the second stage, the adapted backbone is frozen, and a linear hazard probe is trained on the scarce, intervention-verified hazard labels.

GAFT consistently outperforms both frozen DINOv2 and various supervised parameter-efficient fine-tuning (PEFT) baselines. On an intervention-verified forest hazard benchmark, GAFT improved the mean F2 score from 0.0607 (frozen DINOv2) to 0.3757. The authors demonstrate that while supervised PEFT baselines can fit the training scenarios, they fail to generalize to held-out hazard scenarios. In contrast, GAFT's use of geometric inductive bias enables more robust transfer, proving that coupling foundation model adaptation with spatial geometric priors is essential for reliable hazard detection in data-scarce settings.