InfiniVerse: Occupancy Guided Unbounded Scene Generation for Autonomous Driving

Generating realistic, controllable, and temporally coherent urban environments for autonomous driving remains a significant challenge. Existing methods often struggle with long-horizon generation, where errors accumulate over time, leading to geometric drift or visual hallucinations. InfiniVerse aims to solve this by creating a framework that maintains strict 2D-3D alignment and spatial consistency over extended trajectories.

InfiniVerse employs a two-stage pipeline that bridges 3D geometry and 2D video generation. First, it reconstructs a 3D occupancy representation from a single multi-view frame, which serves as a structural foundation. Second, it uses a video diffusion model to translate this occupancy grid into high-fidelity video sequences. The core innovation is a hierarchical sketch-and-refine paradigm: a bidirectional closed-loop mechanism where coarse occupancy grids guide video generation, and the resulting high-fidelity frames are re-projected back into the 3D space to refine the occupancy representation. This mutual constraint ensures that the generated visual appearance and 3D geometry remain aligned, suppressing drift and hallucinations.

InfiniVerse achieves state-of-the-art performance on the Waymo Open Dataset and nuScenes, demonstrating superior generation quality and temporal stability compared to existing benchmarks. It produces long-horizon driving videos that maintain geometric and semantic coherence, with a FID of 6.4 and FVD of 67.97. The bidirectional loop effectively mitigates the common issue of structural collapse in long-term scene generation, allowing for controllable, user-defined trajectory exploration.

This research provides a scalable and controllable paradigm for synthesizing urban environments, which is critical for training and validating perception and planning systems in autonomous driving. By enabling long-range, consistent scene generation from minimal input, InfiniVerse facilitates more robust simulation and data augmentation, addressing a key bottleneck in the development of embodied AI and digital twins.