Pair2Scene: Learning Local Object Relations for Procedural Scene Generation | Xingjian Ran et al. | ResearchPod

Transcript: Pair2Scene: Learning Local Object Relations for Procedural Scene Generation

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a paper called Pair2Scene: Learning Local Object Relations for Procedural Scene Generation. The central puzzle is how to build complex 3D rooms full of furniture that look logical and dense, when most AI systems only learn from sparse data with a few objects per room.

Alex: So why do current systems fail at creating believable, cluttered spaces like a full office?

Sam: Most methods train on datasets averaging just four objects per room. They copy those simple setups, leading to floating items, overlaps, or big empty spots in busier scenes.

Alex: The training data is too sparse for real clutter.

Sam: Real rooms make sense because objects connect to one or two neighbors—like a laptop resting on a table or a chair slid under a desk. The paper learns these local rules from pairs of objects across datasets, then builds the room step by step.

Alex: They start with pairs, like table and laptop?

Sam: Yes. They define support relations for stability, like gravity keeping a laptop on a table, and functional relations for use, like a mouse pad near a computer. Trained on 140,000 pairs from existing scenes, the model uses 3D point clouds—clouds of dots mapping object surfaces—to predict spots for the next object.

Alex: Like snapping Lego bricks together: one connects to another, and the whole structure grows steadily.

Sam: To make the full room work, they check for collisions and gravity, rejecting bad spots. This creates scenes twice as dense as the training data, with over a dozen realistic objects.

Alex: How does it pick the exact spot for something like a keyboard next to a laptop?

Sam: It builds an ordered chain of connections, where each new object links to one already placed. Support links handle bases, like a table under a laptop. Functional links add nearby items that go together, like a keyboard by a laptop.

Alex: Each step chooses a base and figures position from there.

Sam: For precision, it estimates clusters of likely spots—like ways a chair tucks under a desk from left or right—weighted by how common they are. Think of it like plotting safe landing zones on a map.

Alex: How does it combine object shapes with those positions?

Sam: One part pulls shape features, like a table's flat top. Another encodes box sizes and angles. Transformer blocks connect them: self-attention plans the layout, cross-attention matches shapes—like a mug fitting a cupholder. Collision checks and gravity adjustments keep everything stable.

Pair2Scene: Learning Local Object Relations for Procedural Scene Generation

Xingjian Ran, Shujie Zhang, Weipeng Zhong, Li Luo, Bo Dai

Apr 13, 2026·3:42·arxiv·Original Paper

Procedural Scene GenerationLocal Object Relations3D-Pairs DatasetSpatial Reasoning NetworkHierarchical Scene SynthesisPair2Scene

Listen

4 min

Briefing

Overview

Pair2Scene is a breakthrough framework for generating complex, realistic 3D indoor scenes procedurally, overcoming the limitations of data-hungry end-to-end models. Traditional methods train on limited datasets like 3D-Front (avg. ~4 objects/room), producing scenes confined to that narrow distribution. Pair2Scene instead decomposes scene generation into local object relations—physical support (e.g., laptop on table) and semantic functional links (e.g., keyboard near laptop)—modeled as probabilistic rules. These rules are learned from the curated 3D-Pairs dataset and applied recursively within scene hierarchies, using physics simulations and collision-aware sampling for global coherence. The result: diverse, plausible scenes far beyond training data, outperforming diffusion/LLM baselines in spatial logic and complexity.

Local Object Relations

The core intuition: global scene distributions are information-redundant; object placement depends mostly on local dependencies. Pair2Scene captures two key types:

Support Relations (R_s): Physical hierarchies where an anchor object provides a stable surface (e.g., table supports lamp). Modeled via gravitational stability.
Functional Relations (R_f): Semantic utility (e.g., monitor above desk). Conditioned on anchor geometry/position.

Scenes are represented as relational tuples ⟨O_dep, O_sup, {O_fnc}_opt⟩ with causal ordering—anchors instantiate first. This reduces the curse of dimensionality: instead of modeling joint distributions over N objects (exponentially hard), predict dependent positions conditioned on 1-2 anchors.

Spatial Reasoning Network

A transformer-like neural network predicts mixture of logistics (MoL) distributions for dependent object bounding boxes relative to anchors. Key components:

Point Cloud Encoder: Processes 3D point clouds (via Point-MAE) for geometry-aware features.
Bbox Encoder: Captures positions/extents.
Relational Self-Attention: Integrates spatial embeddings into keys/values for global relation capture.
Geometry-Aware Cross-Attention: Fine-grained reasoning over object geometries.

Trained on 3D-Pairs (extracted from existing datasets), it estimates multi-modal position distributions, enabling diverse placements.

Hierarchical Procedural Generation

Inference recursively applies rules:

Start with hierarchy (e.g., room → walls → floor → furniture).
For each dependent object, sample from predicted MoL distribution.
Use collision-aware rejection sampling + physics sims to resolve conflicts.

This bottom-up assembly ensures physical plausibility (no overlaps, stable supports) and semantic coherence (functional groupings). Unlike global autoregressives, it scales to dense scenes (10s of objects) without mode collapse.

Why It Matters

Pair2Scene bridges data scarcity by generalizing beyond distributions—procedural rules compose novel complexity. Evaluations show superior realism vs. baselines like DiffuScene, LayoutVLM on 3D-Front and open-vocab benchmarks. Applications: embodied AI, VR, games needing infinite variety. Future: integrate with asset synthesis for end-to-end scenes.

Abstract

Generating high-fidelity 3D indoor scenes remains a significant challenge due to data scarcity and the complexity of modeling intricate spatial relations. Current methods often struggle to scale beyond training distribution to dense scenes or rely on LLMs/VLMs that lack the ability for precise spatial reasoning. Building on top of the observation that object placement relies mainly on local dependencies instead of information-redundant global distributions, in this paper, we propose Pair2Scene, a novel procedural generation framework that integrates learned local rules with scene hierarchies and physics-based algorithms. These rules mainly capture two types of inter-object relations, namely support relations that follow physical hierarchies, and functional relations that reflect semantic links. We model these rules through a network, which estimates spatial position distributions of dependent objects conditioned on position and geometry of the anchor ones. Accordingly, we curate a dataset 3D-Pairs from existing scene data to train the model. During inference, our framework can generate scenes by recursively applying our model within a hierarchical structure, leveraging collision-aware rejection sampling to align local rules into coherent global layouts. Extensive experiments demonstrate that our framework outperforms existing methods in generating complex environments that go beyond training data while maintaining physical and semantic plausibility.

Pair2Scene: Learning Local Object Relations for Procedural Scene Generation: Research Paper Podcast