Cloning camera motion from reference videos is an important task in video generation, as videos provide intuitive and precise control. Existing methods either directly use parametric representations that fail to handle multi-shot generation or synthesize cross-paired data, which suffer from data scarcity, resulting in poor performance in complicated camera motion cloning. To address these issues, we introduce a general camera motion representation that encodes cameras as grid motion videos. This camera grid represents the camera parameters visually and supports the integration of diverse trajectories for multi-shot video generation. Building upon this, we propose OmniDirector, a unified framework trained on a million-scale camera grid-video pairs that coordinates characters, actions, and cameras to provide director-level control for multimodal diffusion transformers. Furthermore, we design a novel hierarchical prompt expansion agent that harmoniously integrates different control signals by systematically describing camera motion and visual content through understanding signal relationships. Extensive experiments demonstrate the superior performance and outstanding controllability of our framework. Project page: https://ymlinfeng.github.io/OmniDirector.github.io/
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a paper called "OmniDirector," which tackles a specific problem in AI video generation: how do you copy camera movement from one video and apply it to a completely different scene?
Sam: That's the core challenge. The researchers call it "multi-shot camera cloning." Imagine you love the way a camera sweeps and cuts through a film scene—the slow pull-back, the sudden close-up, the dramatic pan. This system tries to capture that sequence of moves and apply it to an entirely new scene, without copying any of the original footage.
Alex: So they're separating the *how the camera moves* from *what the camera is looking at*? Why is that hard?
Sam: Because previous approaches described camera position using rigid numerical coordinates—essentially giving the computer a long list of precise measurements. That kind of description is difficult for an AI to learn from, and it tends to fall apart when you're trying to stitch multiple shots together in sequence.
Alex: If the rigid math is the problem, what's their solution?
Sam: They created what they call a "camera grid." Picture an empty room with a grid of lines drawn on the floor and walls—like a stage set before any actors or furniture arrive. They take the camera's path from a reference video and trace it through this empty space. The result is a kind of motion skeleton—just the movement, with none of the original content attached. It's similar to how a stick figure can show you how someone runs without revealing anything about what that person looks like.
Alex: Oh—so by stripping away the actual video and keeping only the grid, they're turning camera motion into a visual pattern the AI can read directly?
Sam: Exactly. And that matters because AI systems are very good at recognising visual patterns. They trained the model on a dataset of a million of these camera grids, so it could learn the rhythms and shapes of professional cinematography—without ever needing to understand the underlying coordinate mathematics.
Alex: What about the "multi-shot" side of things? How does the model know how to switch between angles without losing the overall flow?
Sam: That's where something they call the "hierarchical prompt expansion agent" comes in. Think of it like a film director who works on two levels at once. At the big-picture level, the director is thinking about how shots relate to each other—when to cut from a wide view to a close-up, how the pacing feels across the whole sequence. At the detailed level, the director is thinking about the specific movement within each individual shot. This system does the same thing: it manages transitions between shots at one level, and the precise motion within each shot at another.
Alex: So it's essentially a two-tiered script for the camera—one layer for the overall structure, one for the fine details?
Sam: That's a good way to put it. And because those two layers are organised separately, the system can keep the scene consistent even when the camera cuts to a new angle. Characters and objects stay where they should be, rather than jumping around unpredictably.
Alex: Can this grid approach handle more complex effects—something like a dolly zoom, where the camera moves forward while the lens zooms out to create that unsettling stretching effect?
Sam: It can. They modify how the grid is rendered to account for the way different lenses distort space. So for a dolly zoom, the grid lines are adjusted to stretch in the way that effect actually looks on screen. They even model fisheye lenses, which curve and warp the image at the edges. The point is that they're not just tracking where the camera is in space—they're simulating the *lens* itself.
Alex: That's a meaningful distinction. The grid isn't just a map of position; it's a map of how the world *looks* through a particular piece of glass.
Sam: Precisely. Which is what makes it a more universal language for camera movement. Different lenses produce very different images from the same position, and this approach accounts for that.
Alex: How does the model then bring everything together—the text description, the grid, and the actual images?
Sam: They use what's called a Multi-Modal Diffusion Transformer. The name sounds technical, but the idea is straightforward. Think of it as a central processing hub that receives several different kinds of information at once—written instructions, the camera grid, and image data—and learns to weigh them against each other. It figures out which signal matters most at any given moment, so the camera movement ends up genuinely aligned with what's happening in the scene, rather than just being pasted on top.
Alex: You mentioned earlier that other systems suffer from "content leakage." How does this approach prevent that?
Sam: Leakage happens when the model gets confused and starts copying visual elements from the reference video into the new scene—backgrounds bleed through, or textures from the original footage appear where they shouldn't. The camera grid solves this cleanly. Because the model only ever sees the lines of the grid—never the original pixels—there's simply no visual data left to leak. The grid acts as a clean intermediary, carrying the movement but none of the content.
Alex: That's an elegant solution. The information bottleneck is the feature, not a limitation.
Sam: That's exactly right. And it's worth noting that the researchers trained the system on a large, diverse dataset specifically to ensure the model learned a wide range of cinematographic styles—not just the most common ones. The goal is a system that gives filmmakers and creators genuine director-level control over AI-generated video, without requiring them to understand the underlying mathematics.
Alex: It's a thoughtful piece of engineering. Taking something as intuitive as "the feeling of a camera move" and finding a way to represent it that a machine can actually work with.
Sam: And doing it in a way that's visually interpretable—you can look at the grid and understand what the camera is doing. That transparency is relatively uncommon in this kind of system.
Alex: Thanks for walking us through it. And thanks to everyone listening to ResearchPod.