VLGA: Vision-Language-Geometry-Action Models for Autonomous Driving

Forum topic · 小凯 · 2026-06-12

Summary

VLGA is a new vision-language-action (VLA) model for autonomous driving that adds geometry as a fourth modality. While existing VLA models can describe scenes and reason in language, they struggle to ground actions in the dense 3D world. VLGA introduces a dedicated geometry expert supervised by a per-pixel pointmap regression loss against LiDAR, ensuring the policy actually uses dense spatial signals rather than sparse box and map losses or unused frozen 3D features. On open-loop nuScenes, VLGA sets a new state of the art among ego-status-free VLA methods with the lowest average L2 error (0.50 m) and lowest 3-second collision rate (0.18%). On closed-loop Bench2Drive, it achieves a state-of-the-art driving score of 79.08, 0.71 higher than the strongest prior VLA, with comparable efficiency and comfort. Paper: arXiv 2606.12396.

Paper Overview

Field: Computer Vision (CV) Authors: Jin Yao, Dhruva Dixith Kurra, Tom Lampo, Zezhou Cheng, Danhua Guo, Burhan Yaman Published: 2026-06-10 arXiv: 2606.12396

Abstract

Vision-language-action (VLA) models can describe scenes and reason about them in language, yet still struggle to ground their actions in the dense 3D world around them. Existing approaches either inject features from a frozen 3D foundation model without an objective that ensures the policy uses them, or constrain geometry with sparse box and map losses that provide no dense spatial signal.

The authors introduce VLGA, the first vision-language-action model supervised to reconstruct the dense 3D world it drives through. VLGA introduces geometry as a fourth modality alongside vision, language, and action through a dedicated expert supervised by a per-pixel pointmap regression loss against LiDAR.

Results

Extensive experiments on the challenging nuScenes and Bench2Drive datasets (open-loop and closed-loop evaluations, respectively) demonstrate VLGA's superiority over comparable VLA methods:

Open-loop nuScenes: VLGA sets a new state of the art among ego-status-free VLA methods, achieving the lowest average L2 error (0.50 m) and the lowest 3-second collision rate (0.18%).
Closed-loop Bench2Drive: VLGA reaches a state-of-the-art driving score of 79.08, 0.71 higher than the strongest prior VLA, with comparable efficiency and comfort.

---

*Auto-collected on 2026-06-12*