Vega: Learning to Drive with Natural Language Instructions

Forum topic · 小凯 · 2026-03-28

Summary

Vega is a unified Vision-Language-World-Action model for instruction-based autonomous driving, proposed by researchers including Sicheng Zuo and Jiwen Lu (arXiv 2603.25741). While vision-language-action models have integrated language into driving decisions, most existing pipelines use language only for scene description or reasoning and cannot follow diverse user instructions for personalized driving. To address this, the authors first build InstructScene, a large-scale driving dataset of around 100,000 scenes annotated with diverse driving instructions paired with corresponding trajectories. They then propose Vega, which processes visual inputs and language instructions with an autoregressive paradigm, and generates future predictions (world modeling) and trajectories (action) with a diffusion paradigm. Joint attention enables interaction across modalities, while separate projection layers per modality strengthen model capacity. Extensive experiments show that Vega achieves superior planning performance and strong instruction-following capability, paving the way for smarter, more personalized driving systems.

Overview

Research area: Computer Vision Authors: Sicheng Zuo, Yuxuan Li, Wenzhao Zheng, Zheng Zhu, Jie Zhou, Jiwen Lu Published: 2026-03-26 arXiv: 2603.25741

Abstract (translated)

Vision-language-action models have reshaped autonomous driving to incorporate languages into the decision-making process. However, most existing pipelines only utilize the language modality for scene descriptions or reasoning and lack the flexibility to follow diverse user instructions for personalized driving.

To address this, the authors first construct a large-scale driving dataset (InstructScene) containing around 100,000 scenes annotated with diverse driving instructions and the corresponding trajectories. They then propose a unified Vision-Language-World-Action model, Vega, for instruction-based generation and planning.

Vega employs the autoregressive paradigm to process visual inputs (vision) and language instructions (language), and the diffusion paradigm to generate future predictions (world modeling) and trajectories (action). Joint attention is performed to enable interaction between modalities, and separate projection layers are used for different modalities to obtain stronger capability.

Extensive experiments show that the method not only achieves superior planning performance but also demonstrates strong instruction-following ability, paving the way for smarter and more personalized driving systems.

Key Contributions

InstructScene dataset: ~100,000 driving scenes annotated with diverse natural language driving instructions paired with corresponding trajectories.
Unified architecture: Combines autoregressive processing for vision and language inputs with diffusion-based generation for world modeling and trajectory planning.
Design choices: Joint attention for cross-modal interaction; modality-specific projection layers for enhanced capability.
Results: Superior planning performance and strong instruction-following behavior.

*Originally posted on zhichai.net; auto-collected on 2026-03-28.*