Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation

How can we improve the precision and generalization of robotic manipulation policies while maintaining a compact, interpretable action representation? The authors address the limitations of traditional 3D Cartesian action spaces, which often suffer from poor spatial grounding and ambiguity, by reformulating high-dimensional 3D control as a more learnable 2D trajectory prediction problem.

Pix2Act projects 3D manipulation actions (gripper poses) onto the image planes of two in-hand cameras as continuous, unbounded 2D keypoint paths. By aligning the action space directly with the visual observation space, the authors introduce a concept called n-equivariance. This allows the model to apply independent, equivariant rotations to both the camera images and their corresponding action trajectories during training. The architecture, termed Diffusion X-Net, uses a multi-view transformer to fuse information from in-hand and global cameras, while a diffusion-based head generates the 2D action trajectories. These are then triangulated back into 3D space for robot execution.

Pix2Act demonstrates superior performance compared to state-of-the-art imitation learning baselines across a variety of simulated tasks, including high-precision assembly and articulated object manipulation. By training on augmented data that respects the geometric symmetry of the camera-action relationship, the policy learns more robust features and generalizes better to unseen scenarios. The approach effectively eliminates the precision loss associated with pixel discretization found in previous 2D-based methods.

This work provides a powerful framework for grounding robotic actions in visual data. By treating the image plane as the primary domain for action generation, the authors leverage the inherent geometric symmetries of robotic manipulation. This not only simplifies the learning task for the neural network but also provides a principled way to incorporate data augmentation that is physically meaningful, leading to more reliable and precise robotic behavior in complex environments.