Gen4U: Unifying Video Generation and Understanding via Diffusion

Can large-scale video diffusion models, typically used for generation, serve as effective, general-purpose encoders for visual understanding tasks? While prior research suggested these models were limited to low-level geometry, this paper investigates whether modern, high-capacity diffusion models have evolved to capture high-level semantics as well.

The authors perform a systematic analysis of the latent spaces of two state-of-the-art video diffusion models: Veo3 and Wan 2.2. Using zero-shot mutual k-Nearest Neighbours (MkNN) alignment metrics, they map how visual representations evolve across network depth and noise levels. By identifying the optimal "sweet spots" for feature extraction, they introduce Gen4U, a framework that repurposes these frozen generative representations for downstream tasks—such as video classification, depth estimation, and captioning—using only a single forward pass.

The study reveals that video diffusion models possess a highly structured latent space. Contrary to earlier beliefs, these models exhibit strong semantic awareness that aligns well with text embeddings and established discriminative encoders. The authors find that while moderate noise levels (around 60%) are optimal for extracting global semantics, lower noise levels contain fine-grained details that are spatially scattered and require attention-based mechanisms to decode effectively. Gen4U demonstrates that these frozen models can achieve state-of-the-art results on benchmarks like Something-Something V2, effectively unifying the paradigms of video generation and perception without requiring expensive fine-tuning or iterative denoising during the understanding phase.

This work challenges the separation between generative and discriminative visual models. By demonstrating that a single, frozen generative backbone can power a wide range of perception tasks, the authors provide a path toward more efficient, multi-purpose foundation models. This approach reduces the need for task-specific training and suggests that the internal representations learned for high-quality video synthesis are inherently rich enough to support complex scene understanding.