Yudong Jin, Tao Xie, Qihang Zhang, Zehong Shen, Zhen Xu, Yujun Shen, Hujun Bao, Xiaowei Zhou, Yinghao Xu
5 min
Abstract
We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as $O(N)$, weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with $O(1)$ reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.
Alex: That's a clean solution. So how do they actually tackle the memory problem when generating all those viewpoints?
Sam: This is where the two main innovations come in. The first is called Reference Context Packing. As the system generates new views, it needs to remember what it's already produced — otherwise each new angle is generated in isolation. But storing every full-sized image from previous rounds would quickly overwhelm memory.
Alex: So what do they do instead?
Sam: They compress past reference views into a fixed-length summary. Think of it like taking detailed notes during a lecture — you don't transcribe every word, you capture the key information in a compact form. The memory cost stays constant no matter how many views have been generated.
Alex: And the second idea?
Sam: That's Target Context Routing, which addresses the drift problem directly. The system generates views in groups, and the trick is in how those groups are arranged at different stages of the process. Early on, when the broad structure of the scene is still being formed, the groups are shuffled so information flows freely across all viewpoints. That prevents any one cluster from developing its own slightly different version of the person's shape.
Alex: And later?
Sam: Once the rough structure is locked in, the groups are fixed into stable clusters. That lets the model settle the fine details — surface texture, sharp edges — without them getting scrambled by cross-group mixing. So mixing early prevents structural drift, and locking later preserves sharpness.
Alex: It's like roughing out a sculpture before you start carving the details.
Sam: That's a fair analogy. The two phases serve genuinely different purposes.
Alex: What are the real-world limits of this approach? Where does it struggle?
Sam: The paper is candid about this. The whole system depends heavily on the quality of the skeleton tracking. If the source video has heavy motion blur, or someone's arm is hidden behind their body for a long stretch, the skeleton estimate degrades — and those errors carry through into the final output. The system is only as reliable as its structural guide.
Alex: What about loose clothing? A flowing coat doesn't follow the skeleton the way a tight-fitting shirt would.
Sam: That's a known limitation the authors acknowledge. When garments move independently from the underlying skeleton — a billowing skirt, a heavy coat — the model can struggle to maintain physical plausibility in the synthesized views. It's an open problem.
Alex: Are there directions they suggest for addressing that?
Sam: They point toward integrating physical simulation for clothing behaviour, and text-driven editing that would let you adjust things like fabric properties or scene lighting directly. Those are future directions rather than current capabilities.
Alex: So taken together, this is a meaningful step toward making volumetric capture — the kind of thing that used to require a room full of cameras — accessible from a single ordinary video. Thanks for walking through the mechanics, Sam.
Sam: Thank you, Alex. And thanks to everyone listening to ResearchPod.