3DCarGen: Scalable 3D Car Generation via 3D-consistent Multi-view Synthesis

Generating high-quality 3D car models from single, real-world images is a significant challenge for autonomous driving and digital twin applications. Existing multi-view diffusion models often struggle with limited viewpoint coverage, cross-view geometric inconsistencies, and difficulties handling the perspective distortions typical of real-world photography. 3DCarGen aims to solve these issues by synthesizing a dense, geometrically consistent set of multi-view images to enable robust 3D mesh reconstruction.

3DCarGen operates in two primary stages. First, it uses a multi-view diffusion model to synthesize dense, 3D-consistent images. To ensure geometric alignment, the authors introduce a "coarse 3D representation" as a synchronization condition. This is achieved by training a multi-view Splatter Image (MV-SI) model that aggregates features from initial views to predict a unified 3D Gaussian Splatting representation. This coarse 3D prior guides the diffusion model, allowing it to generate consistent images from arbitrary viewpoints.

Second, the authors introduce ISOMER+, an extension of the existing ISOMER algorithm. ISOMER+ performs fast mesh reconstruction by incorporating both color and normal map information into a joint optimization process. This ensures that the final mesh geometry is refined using high-fidelity cues, while a one-shot color projection strategy is used to texture the mesh, effectively minimizing the blurry artifacts common in previous generation methods.

Experimental results on synthetic (SRN-Cars, SketchFab-Cars) and real-world (3DRealCar) datasets demonstrate that 3DCarGen outperforms existing methods in both novel view synthesis and final mesh reconstruction quality. By leveraging the coarse 3D prior, the framework maintains geometric consistency across a large number of views, which is critical for producing accurate, high-fidelity 3D assets. The joint optimization of color and normal maps in ISOMER+ further improves surface smoothness and texture clarity compared to methods relying solely on normal maps.