OpenMOSS: In-Context World Modeling for Robotic Control

Modern Vision-Language-Action (VLA) models often struggle to generalize when deployment conditions—such as camera angles or robot morphology—differ from their training data. The authors investigate whether this failure stems from the model's inability to identify the current system configuration at test time, and whether this can be solved by repurposing the model's context window for system identification rather than just task instruction.

The authors introduce In-Context World Modeling (ICWM). Instead of treating the robot's environment as a fixed constant, ICWM forces the model to perform a brief, task-agnostic "probing" phase before executing a task. During this phase, the robot performs random movements and records the resulting visual transitions. These interaction clips are prepended to the model's input as context. By processing these interactions, the model implicitly recovers the latent system configuration (the "world model") and adjusts its subsequent actions accordingly, all without requiring parameter updates or task-specific demonstrations.

ICWM significantly outperforms standard VLA baselines across both simulated benchmarks (LIBERO) and real-world robotic platforms. In simulation, ICWM improved success rates on out-of-distribution viewpoints by 13.0% compared to standard multi-view training. Real-world experiments showed even more dramatic gains, with the model successfully mitigating common failure modes like end-effector drift and premature gripper closure. The authors demonstrate that the model effectively learns to use the interaction context as a dynamic calibration frame, with t-SNE visualizations confirming that the model develops distinct, identifiable internal representations for different system configurations.