Pose6DAug: Physically Plausible Multi-view Object Swapping for Robot Data Augmentation

Vision-Language-Action (VLA) policies often struggle to generalize to novel objects that differ in appearance or geometry from their training data. While collecting more human-led demonstrations is the standard solution, it is prohibitively expensive and time-consuming. This paper asks: can we synthesize high-quality, physically plausible training data for novel objects by repurposing existing successful demonstrations without additional data collection?

The authors introduce Pose6DAug, a framework that performs "failure-driven" data augmentation. Instead of using 2D video editing—which often fails to maintain multi-view consistency or physical realism—Pose6DAug operates in 3D space.

1. Target Analysis: It uses an off-the-shelf model (SAM3D) to reconstruct a 3D mesh of the target object.
2. Trajectory Preservation: It extracts the 6D pose trajectory of the original object from a successful demonstration.
3. 3D-Anchored Swapping: It removes the original object from the video frames using inpainting and re-renders the new target object mesh into the scene. By anchoring the object to the original 6D pose trajectory, the method ensures that the object's motion, grasp contact, and appearance remain geometrically consistent across all camera views.
4. Geometric Perturbations: To increase data diversity, the framework applies physically grounded transformations (rotation, translation, and scaling) to the target mesh before rendering.

Pose6DAug significantly improves the generalization of VLA policies. In experiments on the RoboCasa benchmark, fine-tuning a VLA model with Pose6DAug-augmented data resulted in a 16.5% relative improvement in success rates on novel objects compared to state-of-the-art baselines. The authors demonstrate that by maintaining 3D structural coherence, their method avoids the artifacts common in 2D-only generative editing, leading to more reliable robot behavior in pick-and-place tasks.