Xu Guo, Zhengxuan Wei, Xinghui Li, Hanzhuo Huang, Xinyu Liu, Xiangyang Luo, Min Wei, Yiran Zhu, Qiulin Wang, Yulong Xu, Xintao Wang, Pengfei Wan, Qi Fan, Xiangwang Hou
5 min
Modern video generation models typically treat generation, reference conditioning, and editing as isolated, one-off tasks. This paper introduces Interactive Multi-Shot Video Creation (IMVC), a framework where a single model manages an evolving visual history, allowing users to generate, edit, or reference-guide shots in a continuous, stateful workflow. The core challenge is maintaining long-term consistency across shots while preventing the computational cost of context retrieval from exploding as the video history grows.
ContextMaster addresses the IMVC challenge through three primary innovations:
Role-Aware Context Representation: The model uses specialized rotary coordinates to distinguish between different types of visual inputs—reference images, historical shots, source footage for editing, and the target being generated—ensuring the model understands the distinct semantic role of each.
Cacheable Fixed-Budget Context: To keep inference latency predictable, the model uses a "ConstraintSink" that forces the attention mechanism to prioritize essential reference and source information. It then uses block-sparse attention to dynamically retrieve relevant historical context within a strictly defined, fixed-size budget, ensuring the read cost remains constant regardless of the total number of shots.
Privileged Context Distillation: Because sparse, few-step models often struggle to match the performance of dense, full-context models, the authors employ a two-stage training process. First, a dense teacher model supervises the sparse student via consistency distillation. Second, the student is refined using distribution matching to recover fine-grained perceptual details and correct errors that accumulate over multiple shots.
ContextMaster demonstrates that high-quality, multi-shot video creation is possible within a single, efficient model. By decoupling the context read cost from the history length, the system achieves 16 FPS on a single GPU, making it suitable for interactive, real-time creative workflows where users can iteratively build and refine complex video sequences.
Recent video models increasingly support generation, reference conditioning, and editing within a single model, yet typically expose them as separate operations over fixed inputs. Practical creation unfolds across multiple shots, requiring one model to generate from text, follow a reference, or edit source footage while maintaining shared history. We formalize this setting as interactive multi-shot video creation (IMVC) and introduce ContextMaster, a unified model with a role-aware context representation for these operations. An interactive model must retain access to an expanding history without allowing the context read cost at each denoising step to grow. ContextMaster combines reusable clean context states with fixed budget sparse context routing and uses ConstraintSink to keep task constraints visible. To address the dual challenges of sparse context access and inference with few denoising steps, we propose a two-stage privileged context distillation framework, which transfers full context behavior from a dense teacher through consistency distillation and then refines deployment rollouts with distribution matching. Experiments on the three primitive tasks demonstrate improved task fulfillment and consistency across shots over specialized baselines. User studies further validate flexibly composed workflows, while the model reaches 16 FPS on a single GPU.
Sam: Okay, but here's something I'm curious about — the model is juggling very different kinds of information at once. A reference image of a character is completely different from a video clip or a text edit instruction. How does it keep those straight?
Alex: This is where something called role-aware rotary coordinates comes in. It sounds technical, but the idea is straightforward. Every piece of data that enters the model gets a label — essentially a tag that says "I am a reference image," or "I am a frame from a previous shot," or "I am a new edit instruction." The model uses those tags to understand the role each piece of information plays, rather than treating everything as the same type of input.
Sam: So it's like color-coding your notes. Red for character references, blue for previous scenes, green for new instructions.
Alex: That's a useful way to think about it. Without those labels, the model could easily get confused — treating a reference image as if it were part of the video timeline, or misreading an edit instruction as historical footage.
Sam: Now, if the model is only ever seeing a subset of the full history, doesn't that create a risk? What if it picks the wrong subset and the output quality suffers?
Alex: That's a genuine concern the paper addresses directly. Their solution is a two-stage training approach they call Privileged Context Distillation. In the first stage, a larger "teacher" model is trained with access to the complete history — it sees everything. In the second stage, a faster "student" model is trained to match the teacher's decisions, even though it only has access to the limited budget. The student learns to approximate what the teacher would have done with full information.
Sam: So the student is essentially learning good judgment from someone who has all the facts, even though the student won't always have all the facts themselves.
Alex: That's a precise way to put it. And the result, according to the paper, is a system that maintains output quality close to a full-context model, while keeping the processing cost low enough for interactive use.
Sam: So the real contribution here is making video generation feel less like a one-shot tool and more like a professional editing environment — one that actually understands the history of what you've built.
Alex: That's the central claim. The paper argues that by combining a fixed memory budget, mandatory reference slots, role-aware labeling, and knowledge distillation from a full-context teacher, you can make stateful, multi-shot video creation practical rather than just theoretically possible. Whether that holds up across a wide range of real-world editing tasks is something further work will need to establish, but the framework itself is a meaningful step in that direction.
Sam: It's an interesting shift — from AI as a generator to AI as a collaborator that remembers what you've been building together.
Alex: Well put. Thanks for listening to ResearchPod.