FrozenDrive: Zero-Shot Text-Guided Driving Scene Generation and Data Augmentation with Parameter-Free Frozen Diffusion Model

Autonomous driving models require massive, diverse datasets, yet collecting real-world data for rare scenarios (e.g., adverse weather, night driving) is difficult and expensive. While diffusion models offer a path to synthetic data generation, existing methods often fine-tune the model backbone or add learnable layers to enforce multi-view and temporal consistency. This process frequently erodes the model's pre-trained knowledge, weakening its ability to generalize to unseen prompts or rare conditions. The authors ask: can we achieve high-fidelity, consistent driving scene generation without modifying the pre-trained diffusion backbone?

FrozenDrive introduces a parameter-free architecture that keeps the pre-trained diffusion backbone entirely frozen. Instead of fine-tuning, the authors implement two key mechanisms within the attention pathway:

• Multi-view Inflated Self-Attention (MISA): Concatenates latent features across all camera views to enforce cross-view geometric and semantic alignment.
• Temporal Reference Self-Attention (TRSA): Expands the input context by allowing the current frame to attend to the previous frame, ensuring temporal coherence.

These mechanisms are combined with an object-focused constraint that upweights rare categories during training, ensuring that the model does not prioritize frequent classes at the expense of rare ones. The system is conditioned on structured inputs—HD maps, depth maps, camera indicators, and relative poses—injected via ControlNet.

FrozenDrive successfully synthesizes globally coherent, multi-view driving scenes from text prompts without any weather- or scene-specific fine-tuning. By preserving the original model's pre-trained priors, it demonstrates superior zero-shot performance on unseen text prompts compared to methods that rely on backbone updates. When used to augment the nuScenes dataset, FrozenDrive significantly improves the performance of downstream autonomous driving models, particularly in challenging conditions like night and rain, demonstrating that synthetic data can effectively bridge the gap for rare, hard-to-collect scenarios.