Xiangyu Han, Mengyu Yang, Jiaqi Li, Bowen Chang, Ziyu Chen, Hexu Zhao, Rahul Kumar Agrawal, Anthony Rodriguez, Rajani Acharya, Fiona Hua, Marco Pavone, Chen Feng, Yiming Li
6 min
Abstract
Humans can navigate an unfamiliar city and gradually form a coherent spatial mental map spanning tens of square kilometers. Can AI build spatial representations at a comparable scale? Although recent foundation models have advanced scene reconstruction and embodied intelligence, scaling to entire cities remains an open challenge, primarily due to the lack of city-scale data. To bridge the gap, we introduce WildCity, a real-world multimodal dataset collected by autonomous fleets traversing complex urban environments. Our dataset includes 18 trajectories, each averaging 83.7 kilometers in length, and preserves the core challenges of in-the-wild perception, e.g., dynamic objects, lighting variations, and imperfect camera poses. We further establish an urban-tailored reconstruction baseline and convert the reconstructed environments into a closed-loop simulator. Beyond the dataset and baseline, we systematically analyze the key challenges on the path to simulation-ready urban digital twins: scalability, extrapolation, and uncertainty. Ultimately, WildCity aims to catalyze progress not only in city-scale rendering, but more broadly in the pursuit of AI that can perceive, remember, and reason across space at a scale comparable to human cognition. Project page: https://han-xiangyu.github.io/Wild-City/
Alex: Exactly. To fix this, the paper uses a technique called urban-tailored neural rendering. Rather than just stitching images together, it builds a mathematical model of what the environment should look like—based on thousands of overlapping observations—and forces the geometry to stay consistent with the actual road layout.
Sam: I noticed the paper mentions something called "3D Gaussian Splatting." That sounds like a strange name for something technical.
Alex: It is an unusual name. Here's the idea: instead of representing a scene as a rigid mesh—like a 3D model you might build in a video game—this technique represents it as a cloud of thousands of tiny, flexible shapes. Each shape has a position, a size, and a color. When you want to render a view of the scene, the computer "splats" those shapes onto the screen, like pressing a soft blob of paint. Because the shapes are mathematically adjustable, the system can fine-tune them to match the real world far more precisely than older, rigid methods.
Sam: So the AI isn't just memorizing photographs—it's building a kind of flexible, mathematical model of the city's actual geometry.
Alex: That's the key distinction. And it matters because a system that understands geometry can do something photographs alone cannot: it can predict what a street looks like from a viewpoint that was never in the original footage.
Sam: Is that what the paper calls "view extrapolation"?
Alex: Yes. It's essentially asking: can the model guess what's around a corner it has never seen? If the underlying geometry is solid, the answer is often yes. If the base map is weak, the model hallucinates—filling in gaps with plausible-looking but incorrect details.
Sam: Like a dream where you think you know a place, but the buildings are subtly wrong.
Alex: That's a fair description. Geometric consistency is what separates a reliable model from one that's just guessing convincingly.
Sam: What about moving objects? If a pedestrian walks through the same spot in multiple frames, does the system accidentally bake them into the map permanently?
Alex: That's a real problem, and the paper addresses it with what they call semantic masks. Think of them as digital stencils. Before building the map, the system labels every pixel in every frame—this is a building, this is a road, this is a person. Anything labeled as a moving object gets filtered out before the map is constructed, so only the stable parts of the environment make it through.
Sam: So the city stays clean, and the pedestrians don't become permanent fixtures in the virtual world.
Alex: Exactly. They also use tracking information to distinguish between a parked car—which should stay in the map—and a moving one, which should be removed. It's a meaningful piece of engineering, because a map full of ghost pedestrians would be useless for training.
Sam: Are there any significant limitations the researchers flag?
Alex: They're candid about a couple. The automatic masking works well in most cases, but it still makes mistakes on unusual or rare objects that the labeling system hasn't seen before. And there's a small but persistent issue they call vertical drift—a slight wobbling in the height dimension of the map, at the centimeter scale. It doesn't ruin the map, but it's a known imperfection they note as an area for future work.
Sam: So it's a solid foundation, but not a finished product.
Alex: That's a fair reading. What the paper establishes is a meaningful step: a large-scale, continuous, geometrically consistent dataset that gives researchers the raw material to start asking harder questions—how does an AI remember a path over several kilometers, how does it reason about space it can't directly see, and how do we measure whether it's actually getting better at those things? Those are the questions this dataset is designed to make possible. Thanks for listening to ResearchPod.