We formulate computer vision as unified multimodal generation, where heterogeneous visual tasks are expressed in the native text and image generation spaces of a unified multimodal model, without task-specific architectures. Under this formulation, SenseNova-Vision uses natural-language instructions and optional visual prompts to specify tasks, target regions or views, and decoding conventions, and generates responses as text for symbolic outputs, images for dense spatial predictions, or mixed text-and-image outputs for compositional tasks. To support large-scale training, we convert diverse computer vision annotations into instruction-response examples compatible with these generation spaces, resulting in the SenseNova-Vision Corpus, a computer-vision instruction-response corpus spanning text, image, and mixed targets. Starting from an off-the-shelf pretrained unified multimodal model, SenseNova-Vision is trained primarily on this corpus, with auxiliary multimodal data used as a capability-preserving mixture, and requires no task-specific prediction heads or architectural modifications. The resulting model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation, while supporting language-defined variants that combine category, color, region, and other visual cues. Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry. These results suggest unified multimodal generation as a scalable route for integrating computer vision capabilities into general-purpose foundation models. The model and corpus are publicly available.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper called "Vision as Unified Multimodal Generation." Sam, what's the core idea here?
Sam: So, think about how computer vision works today. If you want an AI to identify objects in a photo, you need one tool. If you want it to estimate how far away things are, you need a different tool. If you want it to outline the exact shape of an object, that's a third tool. The authors of this paper argue that's unnecessarily complicated. Their proposal is to handle all of those tasks with a single model, treating them as one unified problem.
Alex: So rather than a toolbox full of specialised instruments, you'd have one general-purpose system?
Sam: That's the idea. And the way they make it work is by thinking about outputs differently. Most vision systems produce very different kinds of answers—a label, a set of coordinates, a colour-coded map. This model instead asks: can we express all of those answers as either text or an image? Because if so, one system that already knows how to generate text and images can handle everything.
Alex: That's a meaningful shift in framing. How does the model actually decide which format to use?
Sam: Think of it like a chef who can either write out a recipe or plate a finished dish—depending on what you ask for. The model receives a plain-language instruction, something like "detect all the bottles in this photo." If the right answer is a category name or a location, it generates text. If the task needs spatial detail—like painting a precise outline around every object, or producing a depth map that shows which parts of a scene are near and which are far—it generates an image instead.
Alex: And it learns to make that choice on its own? How?
Sam: The researchers built a large training dataset they call the SenseNova-Vision Corpus (/ˈsɛnsnoʊvə viːʒən ˈkɔːrpəs/). The key move was converting all the messy, varied data from different vision tasks into a consistent format: an instruction paired with a response. So the model sees thousands of examples like "here's an image, here's the instruction, here's what the correct output looks like." Over time, it internalises the pattern—segmentation tasks always call for a mask image, detection tasks always call for text with coordinates, and so on.
Alex: So the training data is doing a lot of the heavy lifting. What does that mean for the model's architecture—its internal structure?
Sam: That's actually one of the more significant aspects of this work. Most vision systems bolt on what's called a specialised "head"—a separate module whose only job is to convert the model's internal calculations into a bounding box or a segmentation mask. This model doesn't need that. Because it's already built to generate images and text natively, the same core machinery handles everything. There's no translation layer sitting on top.
Alex: What's the practical advantage of removing that layer?
Sam: It means the model builds shared understanding across tasks rather than learning each one in isolation. When it learns what "depth" looks like from one set of training examples, that knowledge can inform how it handles "segmentation" in another. Specialised systems don't get that cross-pollination—each module only knows its own job.
Alex: Are there trade-offs to this approach?
Sam: Yes, and the authors are candid about it. Because everything depends on a single unified interface, the quality of those instruction-response training examples becomes critical. If the data isn't carefully curated—if the instructions are ambiguous or the responses are inconsistent—the model's performance on demanding tasks like depth estimation can suffer. The flexibility comes with a real responsibility to get the training data right.
Alex: So it's a meaningful step toward making visual AI more programmable and general-purpose, but one that places significant demands on the data behind it. Thanks for walking us through that, Sam, and thanks to our listeners for joining us on ResearchPod.