SceneBind: Binding What and Where Across Vision, Audio and Language

Forum topic · 小凯 · 2026-07-18

Summary

SceneBind is an omni-modal representation framework that jointly captures semantic and 3D spatial understanding across vision, audio, and language. Existing omni-modal encoders excel at instance-level semantics (what is present) but often lack explicit spatial structure (where it is). SceneBind bridges this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots that explicitly capture object-level semantics, spatial attributes, and uncertainty. The authors also propose SceneBind Matching, a semantic-spatial matching scheme integrating global scene similarity with object alignment, enabling cross-modal scene retrieval and object grounding. Training uses a newly curated real-world binaural audio-visual dataset with structured semantic and spatial annotations, plus a cross-modal alignment protocol. Compatible with large-scale pretrained semantic encoders, SceneBind adds only lightweight spatial modeling with a few extra tokens, achieving state-of-the-art scene and spatial retrieval and strong zero-shot transfer to tasks like audio-visual grounding.

Overview

Field: Computer Vision
Authors: Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, Eli Shlizerman
arXiv: 2607.15265

Key Ideas

SceneBind is an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio, and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is).

SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining:

A global semantic embedding for overall scene understanding
Object-centric semantic-spatial slots that explicitly capture object-level semantics, spatial attributes, and uncertainty

SceneBind Matching

The paper further proposes SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting:

Cross-modal scene retrieval
Object grounding

Training and Evaluation

To train and evaluate SceneBind, the authors curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for cross-modal alignment of semantic and spatial signals.

Results

Compatible with large-scale pretrained semantic encoders, adding only lightweight spatial modeling with a small number of extra tokens
Achieves state-of-the-art performance on scene retrieval and spatial retrieval
Strong zero-shot transfer to downstream tasks such as audio-visual grounding

---

*Auto-collected on 2026-07-18.*