We propose Puffin-World, a unified multimodal architecture that integrates physical understanding, spatial simulation, and 3D world generation and reconstruction without relying on external offline modules. To reliably construct and interact with 3D worlds, our framework jointly models three native world states: physics (gravity field and latitude), geometry (depth), and appearance (image), together with a unified Omni-Camera representation that supports diverse tasks and flexible motions. Beyond modeling these states, we introduce a strategy for propagating physical dynamics across future frames. By grounding absolute camera properties in the real world, Puffin-World enables physically consistent and visually stable world generation. We further couple appearance and geometry within a single generative process, jointly synthesizing each future view and reconstructing its underlying geometry. This unified paradigm enables interleaved closed-loop applications requiring synergy across multiple tasks, including mimic and self-calibrated world exploration. To scale Puffin-World to complex scenarios, we construct Puffin-16M, comprising 15 million vision-language-camera triplets and 1 million trajectories featuring various and challenging motions. To foster further research in this area, we released the code, models, and datasets.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a paper that takes aim at a fundamental failure mode in generative world models. Sam, what's the core problem Puffin-World is trying to solve?
Sam: The problem is that current video generation models are essentially stateless with respect to physics. They treat the world as a sequence of 2D images and learn statistical patterns over those images, but they have no persistent representation of physical constraints like gravity. So the moment you ask the camera to move in a non-trivial way, the model starts hallucinating geometry. Horizons tilt, floors curve, walls drift out of vertical.
Alex: Which is a serious problem if you want to use these models as actual simulators — for robotics planning, autonomous driving, anything where geometric consistency matters downstream.
Sam: Exactly. And the failure isn't just aesthetic. If a model can't maintain a stable gravity vector across frames, any downstream policy trained on its outputs is learning from physically incoherent data. The authors frame this as the difference between a video generator and a true world model — and their argument is that the missing ingredient is absolute spatial grounding.
Alex: So what does Puffin-World actually do differently?
Sam: The core contribution is what they call the Omni-Camera representation. Most existing approaches encode camera motion as relative transformations — where the camera moved from the last frame. That's your map. But Puffin-World also encodes absolute orientation — where the camera is pointing relative to the world's gravity axis. That's your compass. The model receives both, concatenated, so it always knows not just how it moved but where it is in an absolute physical sense.
Alex: And that distinction — relative versus absolute — is what prevents the horizon from drifting?
Sam: Right. With only relative motion, the model accumulates error. Each frame's geometry is conditioned on the previous frame, and small inconsistencies compound. What Puffin-World does instead is use physics propagation — the absolute gravity vector is computed once from the reference frame and then propagated forward deterministically to every subsequent frame. The model isn't asked to re-estimate orientation from scratch at each step. It's given a physically consistent anchor and constrained to generate pixels that respect it.
Alex: So the failure mode in prior models wasn't primarily a data problem — it was an architectural one. They were asking the model to infer something it structurally couldn't represent.
Sam: That's the claim, and it's a meaningful reframing. You could throw more data at a relative-motion model and it would still drift on long trajectories, because the information needed to stay grounded simply isn't in the input representation. The gravity vector has to be explicit.
Alex: Where does the training data come in? They introduce Puffin-16M alongside the model.
Sam: Puffin-16M is the dataset they built to support this. The key property isn't just scale — it's rotational diversity. If you train on video data that's mostly forward-facing camera motion, the model never learns to handle large roll or pitch changes, and the gravity-aware representation doesn't get exercised. Puffin-16M is curated to cover a wide range of absolute orientations, which is what lets the model actually learn the constraints the architecture is designed to enforce.
Alex: That's an important design choice. The architecture and the dataset are co-designed — you can't evaluate one without the other.
Sam: Which also means the ablations need to be read carefully. If you remove the gravity-aware field but keep the same training data, you're not just testing the representation — you're also testing whether the dataset's rotational diversity helps a baseline model. Those effects are entangled.
Alex: What does the evidence actually show? Where does the main result sit?
Sam: The load-bearing finding is geometric consistency on long trajectories with significant camera rotation. That's where the gap between Puffin-World and prior video models is largest — models without absolute grounding degrade noticeably as trajectory length increases and rotational complexity grows. The authors also show improvements on novel view synthesis quality, but that's a supporting result. The core claim is about maintaining physical coherence over time, not just rendering quality at a single viewpoint.
Alex: And the principal limitation?
Sam: It's the initialization problem. The entire system depends on accurately estimating the gravity vector at the reference frame. If that initial estimate is wrong — say, from a noisy IMU reading or a failure in the depth estimation pipeline — that error is propagated forward deterministically. The physics propagation that makes the system robust to drift also makes it brittle to initialization error. There's no correction mechanism mid-trajectory.
Alex: So it's a strong architectural solution to one problem that shifts the vulnerability to a different point in the pipeline.
Sam: Precisely. And that's worth flagging for anyone thinking about deployment. In a controlled lab setting with good sensor data, initialization is reliable. In a messier real-world setting — which is exactly where you'd want a robust world model — the quality of that first measurement becomes the binding constraint on everything downstream.
Alex: That's a useful framing for where the work sits. It's a principled solution to geometric drift, with a clear remaining dependency on upstream perception quality. Thanks for walking through it, Sam.
Sam: The underlying idea — that a world model needs an explicit, persistent physical frame, not just learned statistical priors — feels like a durable principle regardless of how the specific architecture evolves.
Alex: Thanks for listening to ResearchPod.