ShotPlan: Cinematic Video Generation with Learnable Planning Token

Modern video diffusion models excel at generating high-fidelity single-shot clips but struggle with the complex requirements of cinematic production, such as maintaining narrative coherence, character identity, and scene consistency across multiple distinct shots. The authors seek to enable explicit, controllable multi-shot video generation within a single, unified denoising process.

The authors introduce ShotPlan, a framework that integrates learnable planning tokens into the video generation process. These tokens serve as temporal anchors that encode transition cues (e.g., hard cuts or cross-fades) and are concatenated with standard visual tokens. To overcome the limitations of temporally compressed latent spaces—where multiple frames map to a single latent index—the authors propose Fractional Temporal Rotary Position Embedding (FRoPE). This allows the model to align transitions with precise, user-specified frame-level timestamps while keeping the original video generation architecture and its pretrained positional priors intact.

ShotPlan significantly outperforms existing multi-shot generation methods in both quantitative and qualitative evaluations. It achieves superior precision in transition timing (lower transition deviation) and demonstrates stronger inter-shot consistency for both characters and scenes. By avoiding architectural modifications like structured attention masks, which can hinder cross-shot information flow, ShotPlan maintains better narrative coherence and allows for flexible, controllable transitions within a single coherent generation pass.

This work bridges the gap between high-quality AI video generation and the practical needs of filmmakers. By providing a mechanism for explicit shot planning, ShotPlan moves video generation models closer to being viable tools for narrative storytelling, allowing creators to organize visual events and guide audience attention through structured, multi-shot sequences.