We present 4DAnyone, a framework for reconstructing 4D humans from an uncalibrated monocular video by generating reconstruction-grade multiview-consistent videos and lifting them into 4D Gaussian Splatting (4DGS). Existing camera-controlled video diffusion models synthesize plausible novel-view videos but fail to maintain consistency when scaled to the tens of target views required for 4DGS reconstruction. We identify this failure as a bounded-attention-context problem: when target views exceed the capacity of a single DiT forward pass, they must be split into groups, exposing two coupled bottlenecks. On the reference-context side, conditioning on all previously generated views grows as $O(N)$, weakening cross-view appearance guidance. On the target-context side, disjoint groups cannot directly exchange information, causing global structural drift. 4DAnyone addresses both bottlenecks with two complementary designs: Reference Context Packing (RCP) compresses growing reference views into a fixed-length mixed-resolution context with $O(1)$ reference-context complexity, while Target Context Routing (TCR) rotates target-view groupings during denoising to share context across groups at high-noise steps and stabilize details at low-noise steps. We further build the MVGameHuman dataset using our in-house game engine and combine it with light-stage and in-the-wild video datasets for training. Experiments on DNA-Rendering and DyMVHumans show that 4DAnyone outperforms prior methods in both novel-view video quality and downstream 4DGS reconstruction, with robust in-the-wild generalization. See our project page for video results and source code: https://4danyone.github.io.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a system called 4DAnyone. The goal is straightforward to describe but genuinely difficult to pull off: take an ordinary phone video of a person, and reconstruct them as a full, interactive 3D model that moves through time.
Alex: So you film someone walking down the street, and the system builds a version of them you could view from any angle?
Sam: That's the ambition, yes. And the core challenge is that existing approaches fall apart when you try to generate the dozens of different viewpoints you'd need for high-quality rendering. They work fine with a handful of angles, but scaling up breaks them.
Alex: Why does scaling break them?
Sam: It comes down to memory. When a computer generates many camera views at once, it runs out of working space and has to chop the job into separate batches. The problem is those batches can't share information with each other, so the different angles start to drift — the arm on the left view doesn't quite match the arm on the right view. The model loses structural consistency.
Alex: Like if you asked different people to each draw one part of a portrait without seeing what the others were drawing.
Sam: That's a good way to put it. So the paper introduces two main ideas to fix this. But before we get there, it's worth understanding how the system figures out where the person's body is in the first place — because that's a surprisingly hard problem from a single video.
Alex: Right — a phone camera doesn't have depth sensors or anything like that.
Sam: Exactly. So instead of trying to estimate precise depth from the video directly — which tends to produce noisy, error-prone results — they use a 3D skeleton. Think of it like a digital stick figure: a set of around forty key points marking the major joints of the body, feet, and limbs. That skeleton gives the system a reliable structural map of where everything is in space.
Alex: But a stick figure misses a lot, doesn't it? Clothing, facial expressions, hair?
Sam: It does, and that's intentional. The skeleton handles the geometry — where the limbs are, how they're oriented. The visual details, the texture of a jacket or the shape of a face, are learned directly from the source video. The two sources of information complement each other.
Alex: So the skeleton is the blueprint, and the video is the paint.
Sam: Exactly. And to make that skeleton more useful, they render it with something called a z-buffer. In plain terms, it's a way of tracking which part of the body is physically closest to the camera. So if an arm crosses in front of the torso, the system knows the arm should block the view behind it. Without that, you get flat, ambiguous poses where the model can't tell what's in front of what.
Alex: That's a clean solution. So how do they actually tackle the memory problem when generating all those viewpoints?
Sam: This is where the two main innovations come in. The first is called Reference Context Packing. As the system generates new views, it needs to remember what it's already produced — otherwise each new angle is generated in isolation. But storing every full-sized image from previous rounds would quickly overwhelm memory.
Alex: So what do they do instead?
Sam: They compress past reference views into a fixed-length summary. Think of it like taking detailed notes during a lecture — you don't transcribe every word, you capture the key information in a compact form. The memory cost stays constant no matter how many views have been generated.
Alex: And the second idea?
Sam: That's Target Context Routing, which addresses the drift problem directly. The system generates views in groups, and the trick is in how those groups are arranged at different stages of the process. Early on, when the broad structure of the scene is still being formed, the groups are shuffled so information flows freely across all viewpoints. That prevents any one cluster from developing its own slightly different version of the person's shape.
Alex: And later?
Sam: Once the rough structure is locked in, the groups are fixed into stable clusters. That lets the model settle the fine details — surface texture, sharp edges — without them getting scrambled by cross-group mixing. So mixing early prevents structural drift, and locking later preserves sharpness.
Alex: It's like roughing out a sculpture before you start carving the details.
Sam: That's a fair analogy. The two phases serve genuinely different purposes.
Alex: What are the real-world limits of this approach? Where does it struggle?
Sam: The paper is candid about this. The whole system depends heavily on the quality of the skeleton tracking. If the source video has heavy motion blur, or someone's arm is hidden behind their body for a long stretch, the skeleton estimate degrades — and those errors carry through into the final output. The system is only as reliable as its structural guide.
Alex: What about loose clothing? A flowing coat doesn't follow the skeleton the way a tight-fitting shirt would.
Sam: That's a known limitation the authors acknowledge. When garments move independently from the underlying skeleton — a billowing skirt, a heavy coat — the model can struggle to maintain physical plausibility in the synthesized views. It's an open problem.
Alex: Are there directions they suggest for addressing that?
Sam: They point toward integrating physical simulation for clothing behaviour, and text-driven editing that would let you adjust things like fabric properties or scene lighting directly. Those are future directions rather than current capabilities.
Alex: So taken together, this is a meaningful step toward making volumetric capture — the kind of thing that used to require a room full of cameras — accessible from a single ordinary video. Thanks for walking through the mechanics, Sam.
Sam: Thank you, Alex. And thanks to everyone listening to ResearchPod.