3D Scene Graph Prediction: Generating Hierarchical Models from Partially Observed Environments

Overview

This paper introduces a hierarchical framework for predicting 3D scene graphs (3DSGs) in indoor environments. While existing research often focuses on object layout generation within a single room, this work addresses the broader challenge of predicting room-level layouts, floor plans, and traversability from partial observations. This capability is critical for robotics applications, such as autonomous exploration and object search, where a robot must anticipate the structure of unexplored areas.

The authors propose a top-down, two-layer generation pipeline:

1. Room-Level Layer: The core contribution is a novel mixed-domain graph diffusion model. It jointly predicts room categories, floor boundaries, and inter-room traversability. By utilizing a masking strategy, the model can condition its predictions on partial floor plans without requiring retraining on partially observed data.
2. Object-Level Layer: The framework integrates an existing mixed discrete-continuous diffusion model (MiDiffusion) to predict object categories, locations, sizes, and orientations within each room, conditioned on the previously generated room geometry.

The room-level model uses a graph transformer backbone to capture pairwise interactions between nodes while maintaining permutation invariance. The geometric features are processed through a latent-space autoencoder, and a post-processing pipeline converts predicted point clouds into clean, polygonal floor plan boundaries.

Findings and Significance

The researchers evaluated their framework on the 3D-FRONT and Matterport3D datasets. The results demonstrate that the proposed method generalizes significantly better to out-of-distribution partial floor plans compared to existing occupancy-based learning baselines. By enabling robots to predict the structure of unobserved regions, this pipeline facilitates more informed decision-making for navigation and exploration tasks. The authors also successfully demonstrated the pipeline on real-world data collected by robots, confirming its practical utility.