SceneBind: Binding What and Where Across Vision, Audio and Language

Forum topic · 小凯 · 2026-07-18

Summary

SceneBind is an omni-modal representation framework that jointly captures semantic understanding (what) and 3D spatial structure (where) across vision, audio, and language. Proposed by Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, and Eli Shlizerman (arXiv:2607.15265), the method addresses a key limitation of existing omni-modal encoders, which excel at instance-level semantics but lack explicit spatial structure. Each scene is represented as a semantic-spatial entity combining a global semantic embedding with object-centric semantic-spatial slots that capture object-level semantics, spatial attributes, and uncertainty. The paper also introduces SceneBind Matching, a scheme integrating global scene similarity with object alignment to enable cross-modal scene retrieval and object grounding. Training uses a newly curated real-world binaural audio-visual dataset with structured semantic and spatial annotations. SceneBind is compatible with large pretrained semantic encoders, adds only lightweight spatial modeling with few extra tokens, and achieves state-of-the-art scene and spatial retrieval with strong zero-shot transfer to tasks like audio-visual grounding.

Paper Overview

Field: Computer Vision
Authors: Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, Eli Shlizerman
Published: 2026-07-16
arXiv: 2607.15265

Abstract

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty.

We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, the authors curated a novel real-world binaural audio-visual dataset containing structured semantic and spatial annotations, together with a training protocol that aligns semantic and spatial signals across modalities.

Key Highlights

SceneBind is compatible with large-scale pretrained semantic encoders, adding only lightweight spatial modeling with a small number of extra tokens.
It achieves state-of-the-art performance on scene retrieval and spatial retrieval benchmarks.
It demonstrates strong zero-shot transfer to downstream tasks such as audio-visual grounding.

---

*Auto-collected on 2026-07-18.*