[论文] SceneBind: Binding What and Where Across Vision, Audio and Language

论文概要

研究领域: CV 作者: Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, Eli Shlizerman 发布时间: 2026-07-16 arXiv: 2607.15265

中文摘要

我们提出SceneBind，一种现实场景的全模态表征，在视觉、音频和语言中联合语义和3D空间理解。现有的全模态编码器擅长实例级语义（即存在什么），但往往缺乏显式空间结构（即在哪里）。SceneBind通过将每个场景表示为语义-空间实体来解决这一差距，结合全局语义嵌入与以对象为中心的语义-空间槽。这种表征显式捕获对象级语义、空间属性和不确定性。我们进一步提出SceneBind匹配，一种语义-空间匹配方案，整合全局场景相似度与对象对齐，支持跨模态场景检索和对象定位。为训练和评估SceneBind，我们整理了一个新颖的现实世界双耳音频-视觉数据集，包含结构化语义和空间标注，并提出了一个跨模态对齐语义和空间信号的训练协议。SceneBind兼容大规模预训练语义编码器，仅增加少量额外token的轻量级空间建模。它在场景和空间检索上达到最先进的性能，同时实现强大的零样本迁移到下游任务，如音频-视觉定位。

原文摘要

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate Scene...

--- *自动采集于 2026-07-18*

#论文 #arXiv #CV #小凯