Vera: A Layered Diffusion Model for Content-Preserving Video Editing

Video editing using diffusion models often struggles with content preservation; because these models typically regenerate the entire video frame-by-frame, they frequently introduce unintended changes to background elements or characters that should remain untouched. The authors investigate whether a layered diffusion approach—where edits are generated as distinct RGBA layers rather than as part of a single composite video—can provide better control and fidelity for professional-grade video editing.

To solve this, the authors introduce Vera, a framework that decomposes the editing process into three distinct outputs: an edit layer, an alpha matte, and a final composite video. By generating these components jointly, the model can explicitly separate new content from the original source. The architecture uses a Mixture-of-Transformers (MoT) design, where three separate Diffusion Transformers (DiTs) handle the edit layer, the alpha matte, and the composite video, respectively. These transformers interact through joint self-attention, allowing the model to maintain spatial and temporal coherence across layers. The authors also curated a large-scale dataset of 486K frames featuring diverse scenes, dynamics, and visual effects to train the model.

Vera significantly outperforms existing open-source video editing models in content preservation metrics (such as PSNR and LPIPS) while remaining highly competitive in overall edit quality and instruction compliance. By explicitly generating an alpha matte, the model effectively isolates the edit from the source video, preventing the "drift" or unintended modifications common in end-to-end diffusion models. The MoT architecture proves effective at handling complex interactions, such as shadows and reflections, which are critical for natural-looking compositing.