Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models (GPD)

Forum topic · 小凯 · 2026-10-12

Summary

GPD (Geometry-Privileged Distillation) is a training method that improves spatial reasoning in vision-language models (VLMs) without requiring 3D inputs at deployment. Spatial errors in VLMs stem from perception—misjudged depth or orientation—that RGB inputs alone cannot correct. GPD exploits the 3D scans already available in spatial training corpora: for each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed, together with the reference answer, to a teacher model within on-policy self-distillation (OPSD). A privileged KL loss is applied only to wrong trajectories, augmenting GRPO, while the deployed model remains pure RGB. On a 4B backbone, GPD achieves 57.1 on VSI-Bench and an average of 37.6 across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming GRPO and answer-privileged OPSD. Ablations confirm the complementarity of 3D and answer privileges, the benefit of question-conditioned routing over full-context injection, and the gains from restricting distillation to incorrect trajectories. Paper: arXiv 2610.12355.

Paper Overview

Field: Computer Vision (CV) Authors: Hongxing Li, Yixin Li, Dingming Li, Zixuan Wang, Yuchen Yan, Wenqi Zhang, Weiming Lu, Yongliang Shen Published: 2026-10-08 arXiv: 2610.12355

Key Points

Problem: Spatial reasoning is a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Spatial errors originate in perception—a misjudged depth or direction can only be corrected by the scene's true geometry.
Existing limitations: Current remedies either inject 3D information into the model at inference time (paying architecture and latency costs) or train with outcome rewards that supervise only the final answer.
Key insight: The 3D-scanned sources of spatial training corpora already provide the ground-truth geometry needed to correct spatial errors—GPD leverages this as a training-time privilege.
Method — GPD (Geometry-Privileged Distillation):
For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text.
These privileged cues, along with the reference answer, are routed to the teacher model within on-policy self-distillation (OPSD).
A privileged KL loss is applied only to wrong trajectories, augmenting GRPO.
The deployed model remains pure RGB—no architectural or latency overhead at inference.
Results (4B backbone):
VSI-Bench: 57.1
Average across MindCube, SPARBench, MMSI-Bench, ViewSpatial: 37.6
Outperforms GRPO and answer-privileged OPSD.
Ablation findings:
3D privileges and answer privileges are complementary.
Question-conditioned routing beats full-context injection.
Restricting distillation to wrong trajectories yields additional gains.

Original Abstract (excerpt)

> Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer...

*Auto-collected on 2026-10-12*