Abdelrhman Werby, Chenguang Huang, Martin Büchner, Abhinav Valada, Wolfram Burgard
4 min
How can robots effectively navigate large-scale, multi-story indoor environments using only natural language instructions? Existing methods often struggle with abstract queries (e.g., 'find the towel in the bathroom on the second floor') or require excessive memory to store dense semantic maps.
The authors present Hierarchical Open-Vocabulary 3D Scene Graphs (HOV-SG). The pipeline operates in two main stages:
To enable navigation, the system constructs a navigational Voronoi graph that connects rooms and floors (including stairs), allowing the robot to plan paths across multiple levels. Queries are parsed by a Large Language Model (LLM) into hierarchical components, which are then matched against the graph nodes using cosine similarity.
HOV-SG outperforms existing open-vocabulary baselines (such as ConceptFusion and ConceptGraphs) in semantic segmentation accuracy on the Replica and ScanNet datasets. In real-world experiments using a Boston Dynamics Spot robot, the system successfully navigated multi-story buildings based on complex, long-horizon language queries. Furthermore, the hierarchical structure provides a 75% reduction in memory footprint compared to dense open-vocabulary maps, making it a highly efficient representation for large-scale environments.
This work bridges the gap between high-level semantic reasoning and low-level robotic navigation. By structuring the environment into a hierarchy that mirrors human conceptualization, HOV-SG allows robots to handle more complex, multi-step instructions in real-world settings without needing massive storage for dense point-cloud features.
Recent open-vocabulary robot mapping methods enrich dense geometric maps with pre-trained visual-language features. While these maps allow for the prediction of point-wise saliency maps when queried for a certain language concept, large-scale environments and abstract queries beyond the object level still pose a considerable hurdle, ultimately limiting language-grounded robotic navigation. In this work, we present HOV-SG, a hierarchical open-vocabulary 3D scene graph mapping approach for language-grounded robot navigation. Leveraging open-vocabulary vision foundation models, we first obtain state-of-the-art open-vocabulary segment-level maps in 3D and subsequently construct a 3D scene graph hierarchy consisting of floor, room, and object concepts, each enriched with open-vocabulary features. Our approach is able to represent multi-story buildings and allows robotic traversal of those using a cross-floor Voronoi graph. HOV-SG is evaluated on three distinct datasets and surpasses previous baselines in open-vocabulary semantic accuracy on the object, room, and floor level while producing a 75% reduction in representation size compared to dense open-vocabulary maps. In order to prove the efficacy and generalization capabilities of HOV-SG, we showcase successful long-horizon language-conditioned robot navigation within real-world multi-storage environments. We provide code and trial video data at http://hovsg.github.io/.
Alex: [probing] But the pipeline is multi-stage. Where does that leave practical deployment? [[RP_SECTION:deployment-constraints-and-limitations|Deployment constraints and limitations]]
Sam: [grounded] That's the critical constraint, and the authors are candid about it. Graph construction happens offline — collect data, build the graph, then query it. That means it assumes a static environment. Move a chair, and the graph is stale. It's a powerful tool for structured, predictable spaces, but not yet a solution for a busy office where the world keeps changing.
Alex: [reflective] And the segmentation dependency compounds that. If the base model mis-segments an object, that error propagates up the tree.
Sam: [direct] That's the deeper bottleneck. Feature aggregation can smooth over noisy embeddings, but it can't recover from a fundamental mis-segmentation. If SAM fails to isolate an object cleanly, that node is unreliable, and any query that routes through it inherits the error. The system's ceiling is set by the quality of its segments — which means it's also hostage to whatever distribution shift exists between the training regime for SAM and the deployment environment.
Alex: [thoughtful] So the honest read is: a meaningful architectural advance for static, multi-floor environments, but one that trades memory and scalability problems for a hard dependency on upstream vision quality and a static-world assumption. [[RP_SECTION:future-research-directions|Future research directions]]
Sam: [professional] That's a fair characterisation. The next frontier the authors point to is dynamic scene graphs that update in real-time — agents that can handle objects being moved, new items introduced, the environment drifting from what was mapped. That's a substantially harder problem. But as a demonstration that language-conditioned, multi-story navigation is computationally feasible at this scale, the hierarchical approach makes a clear case.
Alex: [concluding] A meaningful step forward, with the limitations mapped as carefully as the buildings themselves. Thanks for listening to ResearchPod.