From Perception to Assistance: Open-Vocabulary Shared Autonomy for Robotic Manipulation

How can a mobile robot effectively assist a human operator in performing precise manipulation tasks in cluttered, unstructured industrial environments without requiring complex calibration or pre-defined object models?

The authors propose a shared-autonomy framework that integrates three primary components:

1. Vision-based Teleoperation: A calibration-free interface uses an RGB-D camera to track the operator's body and hand gestures, mapping them directly to the robot's end-effector.
2. Open-Vocabulary Grounding: A vision-language model (Qwen3-VL) and a promptable video-segmentation model (SAM 2) allow the operator to specify targets via natural language. The system then dynamically separates the target object from the surrounding environment in the robot's 3D map.
3. Collision-Aware Control: A GPU-accelerated model-predictive controller (MPC) uses an online volumetric reconstruction (TSDF/ESDF) to enforce safety constraints, while a potential-field layer gently guides the operator's input toward the target during the final approach.

The framework was validated on a quadruped mobile manipulator performing industrial valve manipulation and pick-and-place tasks. The system achieved a positional root-mean-square error (RMSE) of 59 mm compared to ground truth. Crucially, the collision-avoidance module maintained a minimum clearance of 18 cm from obstacles, even when the operator intentionally commanded the arm into them. Ablation studies confirmed that both the collision-avoidance and the potential-field assistance modules were necessary for successful task completion, as removing either led to failures.

This research bridges the gap between flexible human teleoperation and rigid autonomous execution. By allowing operators to use natural language to specify tasks while the robot handles the low-level safety and precision requirements, the framework significantly reduces cognitive load and improves reliability in complex, real-world industrial settings where fully autonomous systems often struggle.