The landscape of AI video generation is undergoing a pivotal shift: moving beyond general generation - which relies on exhaustive prompt-engineering and "cherry-picking" - towards fine-grained, controllable generation and high-fidelity post-processing. In professional AI-assisted filmmaking, it is crucial to perform precise, targeted modifications. A cornerstone of this transition is video instance insertion, which requires inserting a specific instance into existing footage while maintaining scene integrity. Unlike traditional video editing, this task demands several requirements: precise spatial-temporal placement, physically consistent scene interaction, and the faithful preservation of original dynamics - all achieved under minimal user effort. In this paper, we propose PISCO, a video diffusion model for precise video instance insertion with arbitrary sparse keyframe control. PISCO allows users to specify a single keyframe, start-and-end keyframes, or sparse keyframes at arbitrary timestamps, and automatically propagates object appearance, motion, and interaction. To address the severe distribution shift induced by sparse conditioning in pretrained video diffusion models, we introduce Variable-Information Guidance for robust conditioning and Distribution-Preserving Temporal Masking to stabilize temporal generation, together with geometry-aware conditioning for realistic scene adaptation. We further construct PISCO-Bench, a benchmark with verified instance annotations and paired clean background videos, and evaluate performance using both reference-based and reference-free perceptual metrics. Experiments demonstrate that PISCO consistently outperforms strong inpainting and video editing baselines under sparse control, and exhibits clear, monotonic performance improvements as additional control signals are provided. Project page: xiangbogaobarry.github.io/PISCO.
Alex: Welcome to another episode of ResearchPod.
Sam: Imagine a filmmaker wants to add a dragon flying through real city footage they've already shot—at exact spots and times—but without drawing outlines on every single frame or rebuilding the whole scene.
Alex: That's a real challenge in video editing. So either too much work per frame, or the scene gets ruined—like the city suddenly warps unnaturally?
Sam: Exactly. Current tools often demand outlines, called masks, for nearly every frame, which takes hours, or they scramble the background motions entirely. Methods that fill in masked areas keep shapes consistent but can't easily add realistic effects like shadows or ripples from the new object.
Alex: How does this new approach, PISCO, tackle it without all that hassle?
Sam: PISCO is a system built on video generation models that lets users provide just a few snapshots—or keyframes—of the object at chosen moments, like the start or a couple points in time. It then spreads out the object's look, path, and effects across the whole video automatically, while keeping the background's original movements intact. The key is teaching the model to handle these sparse inputs reliably, without glitching.
Alex: So from one or two images, it figures out the full motion and interactions? What makes that propagation work smoothly?
Sam: It starts by training the model to expect varying amounts of guidance info, so it's robust even with minimal keyframes—like practicing a dance with full steps sometimes and just poses other times. This avoids the model getting confused by unusual input patterns that cause flickers or wrong shapes.
Alex: So it gets used to sparse info during training to handle real-world use. But what exactly happens when the inputs are too spotty—does the background start changing anyway?
Sam: Without careful handling, sparse masks confuse the video compressor—a system that squeezes frames into a compact form for processing, like zipping files but for motion across time. They fix this with a technique that blends nearby frame data smoothly in pixel space and adds a signal channel marking what's available, keeping the compression steady. Researchers label this 'Distribution-Preserving Temporal Masking,' or DPTM for short.
Alex: Okay, so it prevents those compression glitches by blending data where needed, without inventing new stuff. And for the dragon to cast real shadows or duck behind cars?
Sam: They layer in depth info from the background video—like a map showing how far each point is from the camera—to sort what's in front or behind. This guides the model to handle overlaps naturally, so the inserted object weaves into the scene without clipping through things. Combined with the sparse training, the background stays exactly as original: same car paths, crowd flows, no warping.
Alex: That preserves the real motions perfectly. How much better does it perform compared to those frame-by-frame masking tools?
Sam: In tests on curated videos, it matched real footage about twice as closely on motion quality metrics versus top editing baselines, and improved steadily with each added keyframe. The paper notes it needs accurate depth estimates and clean backgrounds, so complex scenes with heavy overlaps still challenge it.
Alex: But with all that insertion, how does it keep the original video's motions—like cars driving the exact same paths or crowds moving naturally—completely unchanged?
Sam: The system conditions the generation process on the full original background video itself, so every frame's unaltered motions, shapes, and timings feed directly into the model. It treats the background as fixed input, only modifying pixels where the new object goes, while blending interactions like shadows realistically. This preserves identities and patterns perfectly, unlike prior methods.
Alex: Fixed background as input—got it. What do those prior methods miss that causes issues?
Sam: Take tools like VideoAnydoor, which swap objects between clips but can't pinpoint exact spots or times in long videos without drifting paths. Or Wan-Animate, a base for generating motion from images, yet it lacks tight control over where things go spatially or temporally, often warping backgrounds or losing object details over time. PISCO builds on similar diffusion tech—a process that refines noisy images into clear ones step by step, like sharpening a blurry photo—but adds sparse keyframe conditioning plus depth for precise weaving.
Alex: So diffusion is that noise-to-clear refinement, guided here by background and sparse cues. You've outlined the training—now, how do they test if it actually preserves those original motions better than other tools?
Sam: They built a test set called PISCO-Bench from 100 real videos, hand-picking ones with clear objects to remove and reinsert, ensuring no overlap with training data. For each, they strip out the object to make a clean background, then compare generated insertions back to the originals using metrics that check overall video realism and object quality. Baselines split into three types: agentic pipelines that edit a starting image then expand to video; video inpainting, which fills masked areas across all frames guided by text; and video-to-video editors that tweak whole clips with references and full masks. Most need outlines on every frame or alter backgrounds noticeably.
Alex: So a fair benchmark with ground truth, and dense masks mean way more user work. Does sparse control hold up?
Sam: On whole-video checks, PISCO with first and last keyframes cut the main realism gap from 371 for top baseline VACE to 204—a meaningful drop showing tighter motion match across the scene. In side-by-side checks, its larger version scored highest on keeping the inserted object's look steady over time and held background steadiness competitively. Qualitatively, baselines blurred objects, lost scale or paths over long clips, or warped whole scenes—PISCO kept things precise and natural. The paper cautions resolutions and lengths were fixed for fairness, and dense-mask baselines still edge pixel metrics without sparse flexibility.
Alex: Sparse wins on usability and dynamics preservation—a practical edge for editors.
Sam: Yes—the same setup extends to swaps like changing backgrounds around a fixed object, repositioning or resizing it with real shadows and blocks, tweaking speeds by spacing cues, or simulating new paths.
Alex: Practical extensions for filmmakers or simulators. Any clear limits in the work?
Sam: It relies on clean background videos without the object already there, and solid depth maps for overlaps—errors there cause layering glitches. Tests fixed video length and sharpness for fairness, so scaling to longer or varied clips needs more checks. Complex fast motions with many blocks still improve most from extra keyframes.
Alex: Fair trade-offs: sparse control shines but scales with guidance density. Positions it as a meaningful tool for pro editing—less grind, more control.
Sam: The paper frames it as a step toward easy AI video tweaks: outperforms editing chains on fidelity, gains steadily from sparse frames, and opens doors for VFX without full rebuilds. It's careful work resolving sparsity hurdles in pretrained models.
Alex: A grounded advance. Thanks for joining ResearchPod.