SG-Layout: Structured Scene Graph–Guided Layout Generation with LLMs

Large language models (LLMs) are powerful at semantic reasoning but often struggle with the precise geometric and structural requirements of layout generation. The authors investigate whether explicitly incorporating structured scene graphs—which define objects and their spatial relationships—can bridge the gap between natural language instructions and metrically accurate 2D or 3D layouts.

The authors propose SG-Layout, a framework that treats scene graphs as a primary input modality. The training process follows two stages:

1. Graph-Language Feature Alignment: A relational graph encoder and a projector map scene-graph embeddings into the LLM's latent space, allowing the model to interpret structured spatial data as if it were linguistic tokens.
2. Instruction Tuning: Using LoRA (Low-Rank Adaptation), the model is fine-tuned on layout generation tasks while keeping the backbone LLM frozen. This ensures the model learns to map the aligned graph tokens to specific geometric outputs (e.g., object coordinates, sizes, and orientations) without losing its pretrained general knowledge.

Experimental results across 2D image layout generation, 3D indoor scene synthesis, and robotic object rearrangement demonstrate that SG-Layout outperforms standard LLM baselines. The framework shows significant improvements in spatial reasoning accuracy and geometric consistency. The authors highlight that the performance gains are most pronounced in relation-dense and compositionally complex scenes, where purely text-based models typically fail to ground spatial constraints accurately.

This work provides a scalable, parameter-efficient method for integrating structured spatial priors into LLMs. By converting ambiguous natural language into explicit graph-based guidance, the approach enables more reliable layout generation for applications in computer vision and embodied robotics, where precise object placement is critical.