Mask2Real-WM: Segmentation Masks as a Sim-to-Real Bridge for Controllable Dexterous World Models

Action-conditioned world models are essential for robotic planning and policy evaluation, but building them for high-dimensional dexterous manipulation (e.g., 23 degrees of freedom) is difficult due to the scarcity of large-scale real-world data and the challenge of accurately predicting fine-grained finger-object interactions. The authors introduce Mask2Real-WM, a two-stage model that separates the prediction process into a dynamics model (WM1) and a rendering model (WM2). WM1 predicts future segmentation masks from past masks and action sequences, while WM2 maps these masks to photorealistic RGB images. This decomposition allows the dynamics model to be pretrained on over 50 hours of synthetic simulation data, where the sim-to-real gap is significantly smaller in mask space, and then fine-tuned on less than 2.5 hours of real-world demonstrations.

Mask2Real-WM demonstrates superior action controllability across all 23 degrees of freedom compared to monolithic baselines. By using segmentation masks as a structural intermediate representation, the model achieves high spatial fidelity even when the background or lighting changes. Simulation pretraining is identified as a critical factor; it provides the model with exposure to a wide range of joint configurations and motions that are absent from the limited real-world dataset. Experiments show that while monolithic baselines may capture broad trajectories, they fail to reflect fine-grained, per-joint action effects and suffer from significant performance degradation under out-of-distribution conditions. The two-stage design also offers modularity, allowing the dynamics model to be used as a lightweight checker for future states without the overhead of full RGB rendering.

This work provides a scalable path for training controllable world models for complex robotic tasks where real-world data collection is expensive or dangerous. By using segmentation masks as a bridge, researchers can leverage the vast amount of available synthetic simulation data to learn complex dynamics, while reserving limited real-world data for learning appearance. This approach not only improves the robustness and controllability of robotic world models but also demonstrates that structural representations can effectively mitigate the sim-to-real gap in high-dimensional manipulation tasks.