[论文] DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language ...

论文概要

研究领域: CV
作者: Han Su, Meng Tian, Tianyu Huang, Mingyu Gao, Lennox Wei, Dongxu Li, Jianhua Han, Hang Xu, Xiaohe Wu, Wangmeng Zuo
发布时间: 2026-10-15
arXiv: 2610.22356

中文摘要

视觉语言模型仍难以理解跨视角的 3D 空间关系，也难以在观察者移动时更新空间判断。常见方法将 3D 重建特征融合到视觉 token 中，但在有限空间监督下将这些特征与预训练语义表示对齐仍然困难。我们提出 DK-VLM，采用双核注意力机制，同时从视觉 token 的输入位置和场景中的几何位置计算注意力。语义核保留预训练模型的原生注意力和位置编码，独立的几何核利用估计的 3D 点和相机姿态，通过点位移和相对视图变换直接调制注意力。然后，学习到的门控融合两者的输出。我们还引入参考系组合（RC）监督，训练模型在连续判断中更新参考系。DK-VLM-8B 在六个空间基准上取得最高平均分，在 VSI-Bench（74.33）和 ReVSI-32（58.28）上均获最高总分。项目页面：https://sumuru789.github.io/DK-VLM-website/

原文摘要

Vision-language models still struggle to understand 3D spatial relations across views and update spatial judgments as the observer moves. A common approach fuses 3D reconstruction features into visual tokens, but aligning these features with pretrained semantic representations remains challenging under limited spatial supervision. We propose DK-VLM with Dual-Kernel Attention, which computes attention from both a visual token's input position and its geometric position in the scene. A semantic kernel retains the pretrained model's native attention and positional encoding, while a separate geometric kernel uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations. Then, a learned gate fuses their outputs. Geometry t...

自动采集于 2026-10-17

#论文 #arXiv #CV #小凯