See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

Vision-Language-Action (VLA) models often struggle with camera viewpoint variation because they typically observe scenes in the camera frame while predicting actions in the robot's 3D coordinate frame. This paper investigates how to bridge this frame mismatch to improve generalization across diverse camera setups without requiring complex, non-standard model architectures.

The authors introduce "robot-centric pointmaps," which are images where each pixel stores the 3D coordinates of a scene point relative to the robot base. By lifting RGB-D observations into this robot-centric 3D space, the model receives geometric information that remains consistent regardless of the camera's position. Crucially, because these pointmaps preserve the original H x W image grid, they can be processed by standard 2D visual encoders. The authors use a separate encoder for the pointmap and fuse the resulting features with RGB tokens via element-wise addition, avoiding the need for specialized point cloud encoders or additional token sequences.

Experiments on the RoboCasa benchmark demonstrate that robot-centric pointmaps significantly outperform both standard RGB-only policies and 3D-aware baselines. The authors find that pre-computing robot-frame geometry is more effective than forcing the policy to learn it from camera-frame cues. Furthermore, centering these pointmaps on the robot's end-effector creates a more robust representation, as it clusters interaction targets around a common origin regardless of their absolute position in the workspace. In real-robot tests, the pointmap-based policy maintains high performance even when the camera is moved to placements not seen during training, where RGB-only models typically fail.

This work provides a lightweight, effective way to inject 3D spatial awareness into existing VLA architectures. By leveraging the existing image-based structure of modern foundation models, the approach enables robots to better generalize across varying viewpoints, a critical requirement for deploying manipulation policies in diverse, real-world environments.