Instance-Enriched Semantic Maps for Visual Language Navigation

How can embodied robots effectively balance the need for high-fidelity semantic understanding with the practical constraints of storage and computational efficiency in Visual Language Navigation (VLN)? Existing 2D maps often lose critical vertical information, while 3D scene-graph representations are frequently too storage-intensive for large-scale deployment.

The authors propose a unified framework called Instance-Enriched Semantic Maps. This approach utilizes a 2.5D representation that preserves vertical distinctions by storing multiple instance indices per grid cell, effectively bridging the gap between 2D and 3D methods. The system includes three primary components:

1. Instance-Level 2.5D Mapping: Uses open-vocabulary panoptic segmentation to capture small objects and store diverse semantic attributes, including room-level context and natural language captions.
2. Multi-Type Expert Fusion Retrieval (MTEFR): A query-processing module that routes natural language instructions to specialized experts and fuses their outputs to ensure consistent goal selection across diverse query types.
3. Storage-Efficient Representation: By optimizing how semantic data is stored and associated with spatial grid cells, the system achieves a 96% reduction in storage compared to traditional 3D scene-graph approaches.

The proposed framework demonstrates significant improvements over existing 3D baselines. In navigation experiments, the method achieved over 17% improvement in object retrieval and over 23% in navigation success across various query types. Furthermore, the 2.5D representation outperformed 3D baselines by 27% in prediction-normalized Area Under the Curve (AUC), confirming that the model maintains sufficient spatial fidelity for complex navigation tasks despite its compact storage footprint.