Yitong Xia, Zeren Jiang, Chuanxia Zheng
9 min
Abstract
We introduce Gen4P, a unified framework for recovering 4D geometry and motion from monocular video. We show that a pretrained video generator can be repurposed into a feed-forward 4D perceiver without iterative denoising. Building on on-demand querying, Gen4P processes clean video latents in a single encoder pass and predicts the requested 4D attributes through a lightweight cross-attention decoder. The on-demand query formulation enables flexible spatial sampling, and our proposed continuous-time queries further provide temporal flexibility. To strengthen geometric knowledge, we propose a straightforward plug-in solution by incorporating a pretrained video-depth LoRA as a frozen geometry prior, improving overall performance without adding extra trainable parameters. Gen4P achieves state-of-the-art or competitive performance on tasks of 3D point tracking, video depth and camera pose estimation. Qualitative results on in-the-wild videos illustrate its capability of tracking complex motion while maintaining the delicate structure. Together, these results demonstrate the potential of video generative pretraining for unified 4D perception. See https://gen4p.github.io for more visual results.
Sam: That separates identifying the point from choosing when and where to describe it. Does every new question require processing the video again?
Alex: No. The encoded scene features are reused. A decoder uses cross-attention, meaning the query selects relevant information from those features. It predicts the point’s position and other attributes, including visibility, displacement, and confidence.
Sam: That query design sounds central. Did the authors invent it here, or adapt something established?
Alex: They build on D4RT’s on-demand querying. The change is the video-generator encoder and a different treatment of time. D4RT uses separate learned entries for fixed frame positions. Gen4P represents time by its relative position between the clip’s beginning and end.
Sam: Then a point halfway through a short clip and halfway through a longer clip gets the same relative-time description. How far does that flexibility actually extend?
Alex: Their analysis tests sequences up to 105 frames. Training clips reach only 49 frames. They report no significant performance degradation across the tested lengths, and stable depth error across different query resolutions. That supports flexibility within those tests, not unlimited sequence length.
Sam: Now the generator itself. If it normally works by removing noise, what happens when they stop asking it to generate?
Alex: They use Wan’s pretrained video model as an encoder. Their noise experiments generally favor lower-noise inputs over noisier ones. They then feed it clean video representations and extract features in a single pass, rather than repeatedly denoising.
Sam: But the pretrained model still needs adaptation. How much of it gets retrained?
Alex: The original encoder weights stay frozen. The authors learn low-rank adaptation, or LoRA: restricted updates added to existing weights. They also train the query decoder and prediction head. Together, the main setup has roughly 880 million trainable parameters, so restricted updates do not mean a tiny training job.
Sam: And learning movement doesn’t automatically solve geometry. How do they address the gap they identified between motion knowledge and scene-shape knowledge?
Alex: They add a second adapter, pretrained for video depth estimation. That depth adapter remains frozen while their reconstruction adapter learns alongside it. This imports geometric knowledge without adding trainable parameters relative to their basic variant.
Sam: Does that distinction survive testing? Or would simply giving the reconstruction adapter more capacity achieve the same thing?
Alex: Their ablations favor combining the frozen depth adapter with the learned reconstruction adapter. Widening the learned adapter does not reliably improve overall performance. Fully tuning the generator also worsens several metrics. That supports the specific adaptation design, rather than just adding capacity.
Sam: Let’s get to the main evidence. How broad was the training, and what comparison makes the tracking gain tangible?
Alex: They train on approximately 200,000 clips from eleven public datasets. On Dynamic Replica, their geometry-enhanced version reaches about 93 percent on the dynamic-point tracking score, versus about 82 percent for OpenD4RT. That score averages how often points fall within specified error tolerances; higher is better.
Sam: Is that a general tracking victory, or does it depend on the dataset and which points count?
Alex: It depends. They outperform prior methods clearly on Dynamic Replica and Panoptic Studio, a fully dynamic human–object benchmark. They remain competitive on Aria Digital Twin and PointOdyssey, especially for moving points. Adding the frozen depth adapter improves every reported tracking metric over their basic variant.
Sam: Training and evaluation include some of the same dataset names. Is there a separation between the examples used for learning and testing?
Alex: The authors explicitly address that for Aria Digital Twin, which lacks an official training–test split. They exclude all sequences used by its tracking benchmark. They also evaluate depth on unseen datasets, so that part directly tests transfer beyond their training datasets.
Sam: Does the advantage carry over from tracking into those other measurements?
Alex: Most clearly on Sintel, which has complex camera and character motion. Their depth predictions outperform the compared methods there. Their geometry-enhanced variant also has the lowest Sintel camera-pose errors across all reported pose measures. On the other depth and pose datasets, the picture is competitive rather than uniformly best.
Sam: Before treating these as physical measurements, what qualification would you put closest to those tracking scores?
Alex: The evaluation aligns predictions to ground truth using a scale factor across the sequence. So those scores do not establish accurate absolute physical scale without alignment. That matters if your application needs distances in real units, rather than reconstruction up to scale.
Sam: And a single encoder pass removes repeated denoising, but it doesn’t by itself establish interactive speed. Is there a timing result here?
Alex: The supplied excerpt gives no runtime comparison. It establishes the single-pass design and reuse of features, not a measured real-time guarantee. My reading is that the benchmark evidence supports this as a reconstruction approach, without settling deployment cost.
Sam: Who should read the full document, and where should they start?
Alex: Researchers working on video reconstruction, point tracking, or adapting generative models should read it. Start with World-Coordinate Tracking and its evaluation protocol. Then read Video Generative Encoder and Plug-in Geometry Prior to see what the comparisons actually isolate.
Sam: And what should everyone else remember when this comes up in conversation?
Alex: A model trained to invent moving scenes can also supply useful features for measuring them. Here, that becomes one shared system for tracking, depth, and camera motion—not proof that it wins everywhere.
Sam: Keep the motion results and the scale caveat together.