Humans can navigate an unfamiliar city and gradually form a coherent spatial mental map spanning tens of square kilometers. Can AI build spatial representations at a comparable scale? Although recent foundation models have advanced scene reconstruction and embodied intelligence, scaling to entire cities remains an open challenge, primarily due to the lack of city-scale data. To bridge the gap, we introduce WildCity, a real-world multimodal dataset collected by autonomous fleets traversing complex urban environments. Our dataset includes 18 trajectories, each averaging 83.7 kilometers in length, and preserves the core challenges of in-the-wild perception, e.g., dynamic objects, lighting variations, and imperfect camera poses. We further establish an urban-tailored reconstruction baseline and convert the reconstructed environments into a closed-loop simulator. Beyond the dataset and baseline, we systematically analyze the key challenges on the path to simulation-ready urban digital twins: scalability, extrapolation, and uncertainty. Ultimately, WildCity aims to catalyze progress not only in city-scale rendering, but more broadly in the pursuit of AI that can perceive, remember, and reason across space at a scale comparable to human cognition. Project page: https://han-xiangyu.github.io/Wild-City/
Alex: Welcome to another episode of ResearchPod. Today, we're discussing a new research paper that introduces a dataset called WildCity—and it's asking a genuinely interesting question: can we teach AI to understand a city the way a human does?
Sam: Like, not just recognizing a stop sign, but actually knowing where you are in a city and how everything connects?
Alex: Exactly. The puzzle is whether AI can build a coherent mental map of an entire city, rather than just processing isolated clips. Right now, most AI systems are trained on short, controlled snippets of video. They're good at identifying what's in front of them, but they have no real sense of place.
Sam: So it's like the difference between recognizing a photo of the Eiffel Tower and actually knowing Paris—knowing which street leads where, how far things are, what's around the corner.
Alex: That's a good way to put it. And that gap matters enormously for self-driving cars, delivery robots, or any system that needs to navigate the real world over long distances. The paper argues that to close that gap, AI needs to process thousands of kilometers of continuous, real-world urban data—not just a few carefully curated clips.
Sam: So what makes WildCity different from the dashcam footage that already exists?
Alex: Think of it like the difference between a handful of postcards and actually walking the streets of a city. Existing datasets tend to be short and disconnected—a few minutes here, a few minutes there. WildCity provides continuous streams averaging over two hours per recording, collected across six different cities. That continuity is what lets an AI start to build something resembling a real mental map.
Sam: And the goal is to turn that footage into what the paper calls a "digital twin"?
Alex: Right. A digital twin is a virtual replica of a physical environment—detailed enough that it behaves like the real thing. Think of it as a flight simulator, but for city streets. You can train an AI to navigate there, test how it handles tricky intersections, and see where it fails—all without putting a real vehicle at risk.
Sam: That sounds useful. But I'm guessing turning raw, shaky dashcam footage into a clean 3D replica isn't straightforward.
Alex: Not at all. Imagine trying to stitch together a panorama while moving at thirty miles per hour. The camera vibrates, the lighting shifts from one moment to the next, and the car's GPS position drifts slightly. If your starting measurements are even a little off, those errors compound—and your virtual city ends up subtly warped.
Sam: Like building a house on a foundation that's slightly tilted. Everything looks fine until you realize the walls don't line up.
Alex: Exactly. To fix this, the paper uses a technique called urban-tailored neural rendering. Rather than just stitching images together, it builds a mathematical model of what the environment should look like—based on thousands of overlapping observations—and forces the geometry to stay consistent with the actual road layout.
Sam: I noticed the paper mentions something called "3D Gaussian Splatting." That sounds like a strange name for something technical.
Alex: It is an unusual name. Here's the idea: instead of representing a scene as a rigid mesh—like a 3D model you might build in a video game—this technique represents it as a cloud of thousands of tiny, flexible shapes. Each shape has a position, a size, and a color. When you want to render a view of the scene, the computer "splats" those shapes onto the screen, like pressing a soft blob of paint. Because the shapes are mathematically adjustable, the system can fine-tune them to match the real world far more precisely than older, rigid methods.
Sam: So the AI isn't just memorizing photographs—it's building a kind of flexible, mathematical model of the city's actual geometry.
Alex: That's the key distinction. And it matters because a system that understands geometry can do something photographs alone cannot: it can predict what a street looks like from a viewpoint that was never in the original footage.
Sam: Is that what the paper calls "view extrapolation"?
Alex: Yes. It's essentially asking: can the model guess what's around a corner it has never seen? If the underlying geometry is solid, the answer is often yes. If the base map is weak, the model hallucinates—filling in gaps with plausible-looking but incorrect details.
Sam: Like a dream where you think you know a place, but the buildings are subtly wrong.
Alex: That's a fair description. Geometric consistency is what separates a reliable model from one that's just guessing convincingly.
Sam: What about moving objects? If a pedestrian walks through the same spot in multiple frames, does the system accidentally bake them into the map permanently?
Alex: That's a real problem, and the paper addresses it with what they call semantic masks. Think of them as digital stencils. Before building the map, the system labels every pixel in every frame—this is a building, this is a road, this is a person. Anything labeled as a moving object gets filtered out before the map is constructed, so only the stable parts of the environment make it through.
Sam: So the city stays clean, and the pedestrians don't become permanent fixtures in the virtual world.
Alex: Exactly. They also use tracking information to distinguish between a parked car—which should stay in the map—and a moving one, which should be removed. It's a meaningful piece of engineering, because a map full of ghost pedestrians would be useless for training.
Sam: Are there any significant limitations the researchers flag?
Alex: They're candid about a couple. The automatic masking works well in most cases, but it still makes mistakes on unusual or rare objects that the labeling system hasn't seen before. And there's a small but persistent issue they call vertical drift—a slight wobbling in the height dimension of the map, at the centimeter scale. It doesn't ruin the map, but it's a known imperfection they note as an area for future work.
Sam: So it's a solid foundation, but not a finished product.
Alex: That's a fair reading. What the paper establishes is a meaningful step: a large-scale, continuous, geometrically consistent dataset that gives researchers the raw material to start asking harder questions—how does an AI remember a path over several kilometers, how does it reason about space it can't directly see, and how do we measure whether it's actually getting better at those things? Those are the questions this dataset is designed to make possible. Thanks for listening to ResearchPod.