SceneBind: Binding What and Where Across Vision, Audio and Language

Existing multimodal encoders excel at identifying the semantic content of a scene (what is present) but often struggle with explicit spatial structure (where objects are located). The authors investigate how to unify semantic and 3D spatial understanding across vision, audio, and language to enable better spatial reasoning and cross-modal grounding in real-world environments.

SceneBind represents a scene as a combination of a global semantic embedding and a set of object-centric semantic-spatial slots. Each slot captures object-level semantics, spatial attributes (azimuth, elevation, and distance), and a confidence score. To train this, the authors curated a novel dataset of binaural audio-visual videos with structured spatial annotations. The model uses a two-stage training protocol: first, it performs global semantic alignment using contrastive learning; second, it uses bipartite matching to supervise the object-centric slots, ensuring that predicted slots align with ground-truth object descriptions and spatial locations. At inference, SceneBind Matching combines global scene similarity with object-level alignment to support retrieval and grounding tasks.

SceneBind significantly outperforms existing multimodal encoders like ImageBind and CLAP variants in both scene retrieval and spatial grounding tasks. By explicitly modeling spatial attributes, the framework enables superior zero-shot transfer to downstream tasks such as audio-visual localization, where the model must identify the sounding region in an image. The lightweight design allows it to integrate with large-scale pretrained encoders while adding minimal computational overhead.