Jiwen Liu, Shujuan Li, Zhixue Fang, Xiaohan Li, Yan Zhou, Zijie Meng, Zhimin Zhang, Yawen Luo, Guoxin Zhang, Yu-Shen Liu, Pengfei Wan
5 min
Abstract
Cloning camera motion from reference videos is an important task in video generation, as videos provide intuitive and precise control. Existing methods either directly use parametric representations that fail to handle multi-shot generation or synthesize cross-paired data, which suffer from data scarcity, resulting in poor performance in complicated camera motion cloning. To address these issues, we introduce a general camera motion representation that encodes cameras as grid motion videos. This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation. Building upon this, we propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs that coordinates characters, actions, and cameras to provide director-level control for multimodal diffusion transformers. Furthermore, we design a novel hierarchical prompt expansion agent that harmoniously integrates different control signals by systematically describing camera motion and visual content through understanding signal relationships. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework. Project page: https://ymlinfeng.github.io/OmniDirector.github.io/
Alex: So it's essentially a two-tiered script for the camera—one layer for the overall structure, one for the fine details?
Sam: That's a good way to put it. And because those two layers are organised separately, the system can keep the scene consistent even when the camera cuts to a new angle. Characters and objects stay where they should be, rather than jumping around unpredictably.
Alex: Can this grid approach handle more complex effects—something like a dolly zoom, where the camera moves forward while the lens zooms out to create that unsettling stretching effect?
Sam: It can. They modify how the grid is rendered to account for the way different lenses distort space. So for a dolly zoom, the grid lines are adjusted to stretch in the way that effect actually looks on screen. They even model fisheye lenses, which curve and warp the image at the edges. The point is that they're not just tracking where the camera is in space—they're simulating the *lens* itself.
Alex: That's a meaningful distinction. The grid isn't just a map of position; it's a map of how the world *looks* through a particular piece of glass.
Sam: Precisely. Which is what makes it a more universal language for camera movement. Different lenses produce very different images from the same position, and this approach accounts for that.
Alex: How does the model then bring everything together—the text description, the grid, and the actual images?
Sam: They use what's called a Multi-Modal Diffusion Transformer. The name sounds technical, but the idea is straightforward. Think of it as a central processing hub that receives several different kinds of information at once—written instructions, the camera grid, and image data—and learns to weigh them against each other. It figures out which signal matters most at any given moment, so the camera movement ends up genuinely aligned with what's happening in the scene, rather than just being pasted on top.
Alex: You mentioned earlier that other systems suffer from "content leakage." How does this approach prevent that?
Sam: Leakage happens when the model gets confused and starts copying visual elements from the reference video into the new scene—backgrounds bleed through, or textures from the original footage appear where they shouldn't. The camera grid solves this cleanly. Because the model only ever sees the lines of the grid—never the original pixels—there's simply no visual data left to leak. The grid acts as a clean intermediary, carrying the movement but none of the content.
Alex: That's an elegant solution. The information bottleneck is the feature, not a limitation.
Sam: That's exactly right. And it's worth noting that the researchers trained the system on a large, diverse dataset specifically to ensure the model learned a wide range of cinematographic styles—not just the most common ones. The goal is a system that gives filmmakers and creators genuine director-level control over AI-generated video, without requiring them to understand the underlying mathematics.
Alex: It's a thoughtful piece of engineering. Taking something as intuitive as "the feeling of a camera move" and finding a way to represent it that a machine can actually work with.
Sam: And doing it in a way that's visually interpretable—you can look at the grid and understand what the camera is doing. That transparency is relatively uncommon in this kind of system.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.