Yushi Lan, Zeren Jiang, Koichi Namekata, Xingang Pan, Chuanxia Zheng, Andrea Vedaldi
8 min
Abstract
Synthesizing a monocular video from a new camera trajectory requires preserving the observed scene geometry and motion while completing regions that become newly visible. Feed-forward 4D reconstructors recover the structure of the observed scene, whereas video diffusion models provide a generative prior for completing unseen content. We introduce Holo4D, a video reshooting framework that uses reconstructed 3D trajectories as a holistic source-view 4D scaffold. We project these trajectories into the requested views and encode their geometry and source appearance to guide a pretrained video diffusion model. Together with the source video and target cameras, the scaffold grounds generation in reconstructed observations while leaving unsupported regions to the generative prior. Experiments demonstrate the value of trajectory-based conditioning for camera control and novel-view synthesis. The framework accommodates different 4D reconstructors without retraining the video generator, allowing it to benefit from future advances in source reconstruction. More details can be found in our project page.
Alex: Each trajectory also carries a color sampled from its source observation. The system projects the point's position into each requested camera view. That tells the generator where observed content should appear at the corresponding moment.
Sam: Give me a concrete case where that distinction changes the result.
Alex: In the restaurant example, the camera moves sideways around a diner. Vista4D turns the diner toward the new camera. HOLO4D preserves the diner's orientation while changing the viewpoint. That's the desired separation: move the camera, not the person.
Sam: How does the generator receive those tracks? Is it just shown another rendered video?
Alex: No. A learned encoder converts projected observations into features aligned with the target video. Those features include position, source color, target-camera depth, and validity. The generator receives them alongside the synchronized source video and a separate description of the requested cameras.
Sam: Does that design also change the computational cost, or is the contribution mainly better guidance?
Alex: Both, although the speed claim needs care. The track features attach to existing feature channels rather than adding another sequence for attention to process. Denoising is about a third faster than Vista4D in a matched comparison. But complete run times are similar once loading, conditioning, decoding, and writing are included.
Sam: Let's separate a compelling example from the measured evidence. How did they test whether the camera actually moved correctly?
Alex: On DAVIS, they evaluated a hundred source-video and requested-trajectory pairs. Another reconstruction model recovered cameras from the generated videos. The researchers then compared those recovered paths with the requested ones.
Sam: So camera accuracy is estimated from the outputs, not measured against a captured target video. What did that evaluation find?
Alex: With both methods using the same source reconstruction backend, HOLO4D reduced overall trajectory error by fifty-four percent against Vista4D. Its adjacent-frame translation error was thirty-three percent lower. It also beat the other geometry-rendering baselines on those measures.
Sam: Does it win on rotation too? A camera could move the right distance while looking in the wrong direction.
Alex: On DAVIS, HOLO4D had lower rotation error than the rendered-geometry baselines. The camera-only ReCamMaster was slightly better on rotation, but had the largest overall trajectory error. The authors interpret that combination as consistent with staying close to the source viewpoint.
Sam: What about actual held-out views? Following a camera path doesn't establish that the scene looks correct from there.
Alex: They also tested five iPhone scenes with synchronized source and held-out cameras. Using its two-stage reconstruction backend, HOLO4D led the compared methods on pixel signal fidelity and a learned perceptual-distance measure. But EX-4D led on structural similarity, another image-comparison measure.
Sam: That's useful complementary evidence, though a handful of scenes limits how broadly I'd generalize. Is the reconstruction backend itself doing much of the work?
Alex: It matters substantially. The same trained generator accepts different reconstructors without retraining because the interface uses explicit geometry, not a particular model's hidden features. Changing the reconstruction changes the results, and exact source geometry improves them further.
Sam: That flexibility is attractive, but it also complicates claims of superiority. Does HOLO4D hold its advantage on the competitor's own camera paths?
Alex: Not across the board. On Vista4D's hundred-and-ten-trajectory benchmark, HOLO4D with its default reconstruction improved translation measures but had worse rotation errors. A different backend improved rotation substantially, yet still trailed Vista4D under the source-aligned evaluator. Those comparisons test complete pipelines, not just the generators.
Sam: What evidence shows the generator really uses the trajectories, rather than mainly relying on the source video?
Alex: In a controlled training comparison, removing the color attached to each track raised trajectory error by forty-four percent. The source video was still available. In another test, adding mild noise to otherwise exact tracks tripled rotation error, showing that clean video context didn't compensate for corrupted geometry.
Sam: So the geometric guidance is influential enough to help, but also influential enough to mislead. What happens where there isn't a track at all?
Alex: The generator must infer the content. Large newly revealed regions can look implausible, and fine moving structures such as hands can remain inconsistent. Because inference queries tracks in the first source frame, objects entering later may lack direct geometric support, even though the generator sees them in the source video.
Sam: Given those strengths and limits, who should read the full document, and where should they start?
Alex: Researchers working on controlled video generation or dynamic-scene reconstruction should read the scaffold construction, then the analysis of geometric conditioning. Pay particular attention to the backend substitutions and trajectory-noise tests. For comparison work, read the native Vista4D trajectory evaluation too: it exposes the rotation and alignment caveats.
Sam: And for someone who just needs the central idea, rather than the implementation?
Alex: Track what was seen; generate what wasn't. HOLO4D provides evidence that persistent point paths can anchor a new viewpoint to the original action. It does not turn unseen surfaces into known facts.
Sam: That's the distinction to carry with you.