CG-World: A Large-Scale World-State Dataset for World Model Research

Research Motivation

World models are increasingly expected to move beyond simple video generation toward a deeper understanding of physical dynamics, causality, and control. While existing datasets like Kinetics-700 or Open X-Embodiment provide massive amounts of observational video or robotic trajectories, they often lack the explicit intermediate variables—such as contact graphs, material properties, and internal solver states—necessary to learn the underlying mechanics of the world. CG-World addresses this gap by leveraging industrial computer graphics production pipelines to create a structured, large-scale dataset that aligns visual observations with the explicit physical states that generated them.

Approach and Data Protocol

CG-World introduces a hierarchical data-governance protocol that separates world states from observations. By treating the world as a sequence of states (entities, structures, properties, dynamics, relations, and events) and observations as derived outputs (RGB, depth, flow), the authors enable models to learn the causal mechanisms of the world rather than just the appearance of pixels. The dataset contains approximately 850,000 segments, each 1-5 seconds long, and includes a sophisticated branch-management system. This system categorizes trajectories into factual, interventional, and counterfactual branches, allowing researchers to study how changes in actions or physical mechanisms affect outcomes.

Experimental Results

To validate the utility of the dataset, the authors performed post-training experiments on three tasks: geometry-conditioned video generation, multi-step action prediction, and vision-language-action policy transfer. In video generation, models trained on CG-World supervision showed consistent improvements in metrics like LPIPS and FVD compared to baselines using standard control signals. These results suggest that the structured, multi-modal supervision provided by CG-World helps models better align their internal representations with the physical reality of the scenes they are generating or predicting.

CG-World provides a foundation for training general-purpose world models that are capable of reasoning about interventions and counterfactuals. By bridging the gap between high-fidelity industrial graphics and embodied intelligence, the dataset offers a scalable way to train agents that understand the physical consequences of their actions, potentially reducing the reliance on expensive real-world data collection for early-stage model development.