Do as I Do: Dexterous Manipulation Data from Everyday Human Videos

How can we scalably generate high-quality dexterous manipulation data for robots using the vast, readily available supply of monocular RGB human videos? The authors aim to bridge the gap between observational data (watching humans) and experiential data (robot execution) without relying on specialized hardware like MoCap or depth sensors.

The authors propose a two-stage pipeline: reconstruction and retargeting. First, they use a modular reconstruction process that combines hand tracking with a novel object-tracking method based on guided diffusion. By repurposing a 3D generative foundation model (SAM 3D), they track objects through time by fixing the object's shape and using flow-matching to update its pose, guided by rotational velocity estimates. Second, they perform dynamics-aware retargeting using sampling-based optimization. To handle the inherent noise in reconstructed human trajectories, they introduce three key innovations: warmup steps to stabilize initial states, random force perturbations to ensure robust grasps, and transition rewards to enforce successful object handling during critical phases like picking or placing.

DO AS I DO significantly outperforms existing state-of-the-art methods in both hand-object reconstruction and dexterous retargeting. On a benchmark of 150 in-the-wild internet videos, human evaluators preferred the authors' object tracking 67% of the time compared to existing baselines. In retargeting tasks, the proposed pipeline achieved a 71% success rate on noisy reconstructed references, compared to 25% for the baseline. The authors successfully demonstrated the pipeline on a real-world bimanual robot setup, executing 10 complex tasks including whisking, hammering, and pouring.

This work provides a scalable path for training dexterous robots by leveraging the internet's massive repository of human video data. By removing the need for expensive teleoperation or specialized lab-based data collection, this approach lowers the barrier to entry for teaching robots complex, multi-fingered manipulation skills, ultimately moving the field closer to general-purpose robotic agents.