fARfetch: Enabling Collocated AR-HRC in Large Visually Diverse Environments with VLM-Driven AR Content Adaptation

Research Question and Approach

Human-robot collaboration (HRC) in large, outdoor environments is often hindered by the robot moving beyond the user's visual line of sight (VLOS) and the difficulty of maintaining legible augmented reality (AR) content at long distances. The authors developed fARfetch, a system designed to bridge this gap by integrating three core capabilities: a shared semantic environment map, a context-aware world-in-miniature (WIM) interface, and a vision-language-model (VLM) driven view management system.

By using a Meta Quest 3 headset and a Unitree Go2 quadruped, the system synchronizes semantic landmarks detected by both the robot and the user. These landmarks are embedded into a 3D WIM, allowing users to issue 'go-to' commands or author complex paths by dragging waypoints in a miniature representation of the environment. The VLM-driven view management module dynamically adjusts the color, scale, and orientation of virtual content to ensure it remains legible against complex, changing outdoor backgrounds.

The researchers conducted a within-subjects user study (N=13) involving a 30.5-meter outdoor inspection task. Participants using fARfetch completed tasks 66% faster than those using a non-AR baseline. Furthermore, the system significantly reduced perceived workload, specifically in mental demand (-43%), temporal demand (-34%), and frustration (-66%). A custom legibility survey confirmed that the VLM-driven adaptation effectively maintained the visibility and clarity of virtual content despite the environmental challenges of the outdoor setting.

This research demonstrates that combining shared semantic sensing with intelligent, context-aware AR interfaces can significantly improve the usability of robots in large-scale, real-world environments. By offloading the burden of spatial reasoning and content legibility to a VLM, fARfetch allows users to interact with robots at distances that would otherwise render traditional AR interfaces ineffective. This approach provides a scalable framework for future HRC applications in fields like search and rescue, industrial inspection, and outdoor logistics.