VITAL: From Visual Motion to Precise Action — Vision-Guided Action Tokenizer for End-to-End Driving

Forum topic · 小凯 · 2026-10-16

Summary

VITAL (Vision-Induced Tokens for Action Learning) is an action tokenizer for world-action driving models that derives action targets directly from visual motion instead of trajectory representations defined independently of visual dynamics. The authors first show that reconstructing trajectories from recorded future images captures dominant maneuvers but leaves centimeter-scale errors. VITAL couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold, making visual motion the principal prediction target and providing an independent normalized learning scale for small metric corrections. Using flow matching, the action expert jointly predicts both components from the current image, ego state, and navigation instruction, without generating future images at inference. The same residual coordinates enable bounded reinforcement post-training that refines execution while keeping the scaffold fixed. With a single action denoising step, VITAL reaches 90.8 PDMS on NAVSIM v1 navtest, only 0.13 points below a ten-step score at 3.29x faster end-to-end inference. Without retraining, it achieves 91.1 EPDMS on NAVSIM v2 navtest and 38.6 on navhard. Paper: arXiv 2610.20604.

Paper Overview

Field: Computer Vision (CV) Authors: Ao Liang, Youquan Liu, Mingxiao Li, Wei Chow, Lingdong Kong Published: 2026-10-14 arXiv: 2610.20604

Abstract

World-action models aim to turn visual anticipation into driving decisions, yet typically train action experts on trajectory representations defined independently of visual dynamics. This motivates deriving action targets directly from visual motion.

The authors first examine a simpler setting in which trajectories are reconstructed from recorded future images. Even without the need for future prediction, reconstruction captures the dominant maneuver but leaves centimeter-scale errors.

They therefore introduce VITAL (Vision-Induced Tokens for Action Learning), an action tokenizer that couples a visually derived motion scaffold with a compact residual encoding the trajectory error left by that scaffold. This representation makes visual motion the principal prediction target and gives small metric corrections an independent, normalized learning scale.

Method Highlights

Dual-component tokenization: motion scaffold derived from visual dynamics + compact residual for metric refinement.
Flow matching action expert: jointly predicts both components from the current image, ego state, and navigation instruction — no future image generation required at inference.
Bounded reinforcement post-training: the same residual coordinates support refining execution while keeping the predicted visual scaffold fixed.

Results

NAVSIM v1 navtest: 90.8 PDMS with only a single action denoising step — just 0.13 points below a ten-step baseline, at 3.29x faster end-to-end inference.
NAVSIM v2 navtest: 91.1 EPDMS, without any retraining.
navhard: 38.6.