360CityArena: A Realistic Virtual Urban Navigation Benchmark for Embodied Agents

360CityArena addresses a critical gap in embodied AI: the lack of benchmarks that combine photorealism with the complexity of real-world urban navigation. While existing simulators often rely on synthetic 3D assets or disconnected panoramic images, 360CityArena uses a pose-graph reconstruction of the Akihabara district in Tokyo, built from 602 interconnected 360-degree video segments. This allows for continuous, egocentric navigation through a dense, information-rich urban environment.

The authors developed 175 human-crafted tasks categorized into three domains: Environment Understanding (localization and landmark search), Path Reasoning (map-based navigation and instruction following), and Spatial Reasoning (relational understanding and object counting). By using a consistent environment and standardized evaluation protocols—including exact, fuzzy, and coordinate-based matching—the authors provide a rigorous testbed for assessing how well AI agents perceive, reason, and act in a city-scale setting.

Evaluation of state-of-the-art large multimodal models (LMMs) demonstrates a substantial performance gap between current AI and human capabilities. Even the strongest tested model, Gemini 2.5 Flash, achieved only 17.1% accuracy compared to the human baseline of 77.3%. The study also reveals that providing explicit map-based location priors does not consistently improve performance, suggesting that current models struggle to effectively integrate global spatial information with local visual observations.

As embodied agents move from controlled indoor environments to the unpredictable, visually complex outdoors, benchmarks like 360CityArena are essential for measuring progress. The findings highlight that current LMMs, despite their impressive reasoning capabilities in other domains, are not yet ready for reliable city-scale navigation, pointing toward a need for better integration of spatial awareness and long-horizon planning.