DK-VLM: Geometry as Position for Spatial Reasoning in Vision-Language Models

Field: Computer Vision (CV) Authors: Han Su, Meng Tian, Tianyu Huang, Mingyu Gao, Lennox Wei, Dongxu Li, Jianhua Han, Hang Xu, Xiaohe Wu, Wangmeng Zuo arXiv: 2610.22356 Project page: https://sumuru789.github.io/DK-VLM-website/

Problem

Vision-language models still struggle to understand 3D spatial relations across views and to update spatial judgments as the observer moves. A common approach fuses 3D reconstruction features into visual tokens, but aligning these features with pretrained semantic representations remains challenging under limited spatial supervision.

Method

DK-VLM introduces Dual-Kernel Attention, which computes attention from both a visual token's input position and its geometric position in the scene:

• Semantic kernel: retains the pretrained model's native attention and positional encoding.
• Geometric kernel: uses estimated 3D points and camera poses to directly modulate attention through point displacements and relative view transformations.
• A learned gate fuses the outputs of the two kernels.

Additionally, the paper proposes Reference-frame Composition (RC) supervision, which trains the model to update its reference frame across consecutive spatial judgments as the observer moves.

Results

• DK-VLM-8B achieves the highest average score across six spatial benchmarks.
• Top total scores on VSI-Bench (74.33) and ReVSI-32 (58.28).

Links

• Paper: https://arxiv.org/abs/2610.22356
• Project page: https://sumuru789.github.io/DK-VLM-website/