ResearchPod Summary
How can we automatically construct inspectable, simulator-style world models that are both scalable and effective for planning? Existing neural world models are often opaque, while hand-built physics engines are difficult to scale. This paper explores whether executable code can serve as a structured latent representation that bridges the gap between these two approaches.
VisualPatchWorld (VPW) introduces a two-level induction process to recover dynamics from interaction data. First, the system performs 'active probing'—running short, discriminating experiments in the environment—to select the correct qualitative dynamical sketch (e.g., contact-driven vs. action-driven). Second, it fits the free parameters of the chosen Python template by minimizing multi-step prediction error over recorded state-action traces. The resulting programs are fully executable, allowing them to be rolled forward like a simulator and used within a model-predictive control (MPC) framework to evaluate candidate action sequences.
VPW demonstrates that structured code induction is a viable path for planning. Across four control domains, VPW achieved a 69.0% mean planning success rate, outperforming the strongest existing code-based baseline by 23.5 percentage points. The model's performance is particularly strong in navigation and grasp-rich control, where it approaches the success rates of ground-truth physics engines. While a performance gap remains in contact-rich pushing, the authors show that re-scoring a small shortlist of promising plans using a ground-truth engine effectively bridges this gap, suggesting that the induced model is highly effective at pruning the search space.
This work provides a practical methodology for creating 'white-box' world models. Because the dynamics are represented as source code, they are inherently inspectable and editable, offering a significant advantage over neural latent models when a prediction fails. By automating the selection of dynamical structure, VPW moves closer to the goal of building robust, general-purpose simulators directly from interaction data.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.