Letian Wang, Chuhan Zhang, Rishabh Kabra, Jasper Uijlings, Steven Waslander, Andrew Zisserman, Joao Carreira, Kaiming He, Misha Andriluka, Eduard Gabriel Bazavan, Andrei Zanfir, Cristian Sminchisescu
6 min
Computer vision has long relied on task-specific models, such as separate architectures for depth estimation, segmentation, or pose tracking. This paper investigates whether the paradigm shift seen in Natural Language Processing—moving from specialized models to unified foundation models via next-token prediction—can be replicated in vision. The authors propose that large-scale text-to-video generation serves as the necessary catalyst, providing the spatiotemporal priors and vision-language alignment required for general-purpose visual intelligence.
The authors introduce GenCeption, a unified perception model built upon a pre-trained video generative diffusion backbone. Instead of using the slow, iterative denoising process typical of generative models, GenCeption is reformulated as a single-step, feed-forward architecture. The model is fine-tuned on a diverse suite of tasks, including depth, surface normal, and camera pose estimation, using predominantly synthetic data. By treating these tasks as sequence-to-sequence mapping problems steered by text instructions, the model achieves a unified approach where task specifications are defined by data format rather than architectural changes.
GenCeption achieves state-of-the-art performance across a wide range of dense and sparse vision tasks, often matching or outperforming specialized models like DepthAnything3 and SAM3. The model demonstrates significant data efficiency, reaching performance levels comparable to leading specialized models while using 7 to 500 times less training data. Furthermore, the authors observe emergent behaviors, such as the ability to generalize from synthetic human-centric training data to real-world footage and out-of-distribution object categories like animals and robots, suggesting that the generative backbone successfully internalizes universal physical and geometric priors.
This work suggests that video generation is not merely a tool for creating synthetic media, but a foundational path toward generalist vision intelligence. By proving that a single, unified architecture can handle heterogeneous perception tasks, the authors provide a scalable roadmap for computer vision that mirrors the success of Large Language Models, potentially collapsing the need for disparate, task-specific vision systems.
Driven by next-token prediction, NLP shifted from task-specific models into powerful generalist foundation models. What, then, is the equivalent catalyst needed to achieve a general-purpose model in computer vision? In this paper, we contend that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision, providing the necessary spatiotemporal priors, vision-language alignment, and scalability required for general visual intelligence. We introduce GenCeption, which leverages a pre-trained video generative diffusion backbone to define a feed-forward perception model, capable of performing various vision tasks steered by text instructions. Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks, including depth, surface normal, and camera pose estimation, expression-referring segmentation, and 3D keypoint prediction, often matching or surpassing specialized models (e.g. DepthAnything3, SAM3, D4RT, VGGT-Omega, Sapiens, David, Genmo, and Lotus-2). Furthermore, the video generative pretrained backbone outperforms alternative pretraining paradigms (e.g., V-JEPA, and Video MAE) under comparable settings. Importantly, GenCeption exhibits preliminary data and model scaling properties along with exceptional data efficiency, where it achieves comparable performance with leading models like D4RT and VGGT-Omega with 7 to 500 less training data. Finally, GenCeption also exhibits intriguing emergent behaviors: a model trained exclusively on synthetic human videos generalizes to real-world footage and out-of-distribution object categories (e.g., animals and robots). These findings suggest that video generation is not merely a synthesis tool, but a foundational path toward generalist vision intelligence for the physical world. Project page: https://genception.github.io
Sam: Here's what I don't immediately understand, though. Generating video is usually a slow process—the model refines its output step by step, like an artist sketching and erasing repeatedly. How do they make that fast enough to be practical?
Alex: That's the core technical contribution of the paper. Normally, these generation models work iteratively—they make a rough attempt, check it, adjust, check again, dozens of times over. The researchers collapse that entire process into a single pass. They call this "feed-forward diffusion." Instead of going back and forth to refine its output, the model looks at the input once and produces the result directly.
Sam: So it's the difference between a painter who does thirty drafts and one who gets it right on the first stroke?
Alex: That's a fair way to put it. They achieve this using a technique called "rectified flow," which creates a direct, straight-line path from the input video to the output—whether that output is a depth map, a segmentation, or a pose estimate. No iterative refinement required.
Sam: And does this actually hold up? Does a generalist system like this perform as well as the specialized models that were built specifically for each task?
Alex: The paper suggests it does. GenCeption matches or outperforms specialized models on tasks like depth estimation and pose tracking, and it often does so using significantly less training data. The paper is careful not to overstate this—these are results on specific benchmarks—but the pattern is consistent across the tasks they tested.
Sam: What I find notable is the generalization. The model was trained mostly on human video, but it apparently works on footage of animals and robots too?
Alex: Yes, and that's arguably the most telling result. The model was never shown a robot during training, yet when given robot footage, it could still estimate depth and track motion correctly. That suggests it's not simply memorizing patterns from its training data—it has learned something more general about how physical objects behave in the world.
Sam: So the practical implication is that you don't need to build and train a new model every time you want to add a new capability. You just give this one model a different instruction?
Alex: That's the goal. By treating vision tasks as prompts—as instructions rather than hardwired behaviors—the model becomes a generalist. You tell it what you need, and it shifts its focus accordingly, drawing on the same underlying world model each time. It's a bit like the difference between a Swiss Army knife and a drawer full of single-purpose tools.
Sam: It reframes the whole problem, doesn't it? Instead of asking "how do we teach a machine to detect depth," the question becomes "how do we teach a machine to understand the world well enough that depth detection is just one natural consequence."
Alex: That's precisely the shift the paper is proposing. When you train a model to simulate reality convincingly, it has no choice but to learn the structure of reality. The perception tasks become almost secondary—outputs of a deeper understanding that was built in from the start.
Sam: It's a meaningful step, and one that points toward a different philosophy for how we build visual AI—less about solving individual problems, more about building systems that genuinely understand the world they're looking at.
Alex: Well put. The paper is still early-stage research, and there are open questions about how well this scales and how it performs across a wider range of real-world conditions. But the direction is clear, and the early results are worth paying attention to. Thanks for listening to ResearchPod.