Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a framework called Image2Sim—and it tackles a real bottleneck in how we train robots to move through the world.
Sam: So the core problem is that robots need a lot of practice, but we don't have enough realistic spaces to practice them in?
Alex: Exactly. Think about how a child learns to move around their house. They bump into things, they explore, they gradually build up a sense of the space. Robots need that same kind of experience—but right now, giving them enough of it is genuinely difficult. You can build environments by hand, but hiring people to model every room, every hallway, every kitchen is slow and expensive. And if you cut corners and use simplified, artificial spaces, the robot often struggles when it finally meets the real world.
Sam: So there's a gap between the training environment and reality. The robot learns one thing and then encounters something quite different.
Alex: Right. Image2Sim tries to close that gap by starting with something we already have in abundance: ordinary video footage and photographs. Instead of building environments from scratch, it converts that existing visual material into tens of thousands of interactive 3D spaces a robot can actually move through.
Sam: But how does a flat video become a 3D space with real depth and structure?
Alex: It works in two stages. The easiest way to picture it is an artist at work. The first stage is like a sketch artist. It looks at the video and figures out the physical structure of the room—where the walls are, how far away the furniture is, what the floor plan looks like. The technical name for this is a Feature Gaussian model, but the idea is simple: it's building the skeleton of the space.
Sam: So the sketch gives you the bones of the room. What does the second stage add?
Alex: The second stage is the painter. Once the skeleton exists, it fills in everything that makes the room look real—the textures on the walls, the way light falls across a table, the colour of the carpet. The tricky part is that a camera never captures every angle. There are always corners it didn't see. So this stage has to intelligently fill in those gaps—and it uses the geometry from the first stage as a guide. Because it already knows where the walls actually are, it's not guessing blindly. It's making informed decisions about what those unseen areas probably look like.
Sam: So the structure prevents the system from inventing things that couldn't physically be there.
Alex: Precisely. That's why keeping the two stages separate matters. Systems that try to do everything at once tend to produce what researchers call hallucinations—realistic-looking details that are physically wrong, like a doorway appearing in the middle of a solid wall. By handling structure and appearance separately, Image2Sim stays grounded in the actual geometry of the space.
Sam: So now you have a realistic, physically accurate 3D room. But a robot doesn't just need a map—it needs to know what to do inside it. How does the system handle that side of things?
Alex: This is where what the paper calls the embodied data engine comes in. Once the environment exists, the system needs to populate it with tasks—actual things for the robot to practice. It uses a motion planning system to calculate valid routes through the space. Think of it like a GPS, but instead of roads, it's mapping floors, doorways, and corridors. And crucially, it assigns a higher penalty to risky paths—tight corners, cluttered areas—so the robot is nudged toward developing safer movement habits from the start.
Sam: It's teaching the robot to prefer the sensible route, the way a careful person would choose a wide hallway over squeezing past furniture.
Alex: That's a good way to put it. And then a language model looks at those planned paths and describes them in plain English—breaking a journey into steps and writing out instructions like "go to the kitchen" or "move toward the front door." So the system is automatically generating both the practice run and the verbal instruction that goes with it. At scale, this produces millions of training examples without a human writing a single one.
Sam: And the question that matters most: does any of this actually transfer? If a robot trains in these generated environments, does it hold up in a real building?
Alex: The study suggests it does. Models trained using Image2Sim show meaningful improvements in navigating real-world settings, and notably, they don't require additional fine-tuning once they're deployed. That said, the paper is clear about the current limits. Because the system prioritises speed and scale, it's not yet well-suited to fine-grained physical interactions—picking up objects, for instance, or navigating around people who are moving through the space.
Sam: So it's a solid foundation for navigation, but more complex tasks—like fetching something from another room—are still ahead.
Alex: That's where the research points next. The authors suggest future work will look at integrating object manipulation and more dynamic environments—moving toward multi-step instructions in any home setting. But as a step toward making robot training scalable without requiring enormous human effort, this framework represents a meaningful advance. The underlying insight is worth holding onto: we're surrounded by video of the real world, and this paper asks whether that footage can do the work that hand-built environments currently can't.
Sam: A practical approach to a problem that's been holding the field back. Thanks for walking me through it, Alex.
Alex: Thanks for listening to ResearchPod.