Zihan Wang, Seungjun Lee, Yinghao Xu, Gim Hee Lee
5 min
Abstract
Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.
Sam: So now you have a realistic, physically accurate 3D room. But a robot doesn't just need a map—it needs to know what to do inside it. How does the system handle that side of things?
Alex: This is where what the paper calls the embodied data engine comes in. Once the environment exists, the system needs to populate it with tasks—actual things for the robot to practice. It uses a motion planning system to calculate valid routes through the space. Think of it like a GPS, but instead of roads, it's mapping floors, doorways, and corridors. And crucially, it assigns a higher penalty to risky paths—tight corners, cluttered areas—so the robot is nudged toward developing safer movement habits from the start.
Sam: It's teaching the robot to prefer the sensible route, the way a careful person would choose a wide hallway over squeezing past furniture.
Alex: That's a good way to put it. And then a language model looks at those planned paths and describes them in plain English—breaking a journey into steps and writing out instructions like "go to the kitchen" or "move toward the front door." So the system is automatically generating both the practice run and the verbal instruction that goes with it. At scale, this produces millions of training examples without a human writing a single one.
Sam: And the question that matters most: does any of this actually transfer? If a robot trains in these generated environments, does it hold up in a real building?
Alex: The study suggests it does. Models trained using Image2Sim show meaningful improvements in navigating real-world settings, and notably, they don't require additional fine-tuning once they're deployed. That said, the paper is clear about the current limits. Because the system prioritises speed and scale, it's not yet well-suited to fine-grained physical interactions—picking up objects, for instance, or navigating around people who are moving through the space.
Sam: So it's a solid foundation for navigation, but more complex tasks—like fetching something from another room—are still ahead.
Alex: That's where the research points next. The authors suggest future work will look at integrating object manipulation and more dynamic environments—moving toward multi-step instructions in any home setting. But as a step toward making robot training scalable without requiring enormous human effort, this framework represents a meaningful advance. The underlying insight is worth holding onto: we're surrounded by video of the real world, and this paper asks whether that footage can do the work that hand-built environments currently can't.
Sam: A practical approach to a problem that's been holding the field back. Thanks for walking me through it, Alex.
Alex: Thanks for listening to ResearchPod.