LaST-VLA: Physically Grounded Latent Spatio-Temporal Reasoning for Autonomous Driving VLA Models
> TL;DR: LaST-VLA (Tsinghua + Xiaomi + University of Macau) replaces textual chain-of-thought with a continuous latent spatio-temporal reasoning space aligned with physical geometry and dynamics, achieving a record 91.3 PDMS on NAVSIM v1.
The Problem: Why Textual Chain-of-Thought Falls Short
Vision-Language-Action (VLA) models are unifying perception and planning in autonomous driving, but language is a coarse approximation of physical space. Textual CoT suffers from three key weaknesses:
| Issue | Manifestation | Consequence | |-------|---------------|-------------| | Semantics-perception decoupling | Descriptions drift from visual evidence | Planner follows wrong linguistic guidance | | Semantic hallucination | Text describes scenes inconsistently with reality | Dangerous planning errors | | High latency | Long intermediate text generation | Inference cost surge, overthinking |
Naive latent CoT avoids the text bottleneck but has a fatal flaw: without explicit intermediate constraints, latent reasoning is not grounded in physical laws—no notion of distance, speed, or collision probability.
The Solution: Physics-Grounded Latent Reasoning
Dual Feature Alignment
LaST-VLA distills physical knowledge from two foundation models:
- 3D geometry foundation model (Cosmos/VGGT): spatial constraints (distance, orientation, boundaries), distilled via a geometry adapter into a geometry channel H_g
- Video world model: dynamic foresight (future motion prediction), distilled via a dynamics adapter into a dynamics channel H_d
- Mutual masks: 3D and world-model tokens are isolated from each other
- Visual bottleneck mask: action tokens cannot attend directly to raw images—all visual information must pass through latent reasoning
- Geometry only: 90.1 PDMS; Dynamics only: 90.0; Both: 91.3 (baseline without latent CoT: 87.9)
- Unsupervised latent CoT: 89.8 (beats textual CoT at 87.2); supervised latent CoT: 91.3—physical alignment is the decisive factor
- Validated mainly on NAVSIM (OpenScene-derived); real-world generalization needs further work
- Latent reasoning is less interpretable than textual CoT
- Quality of teacher models (3D and video world models) directly affects distillation
- Paper: *LaST-VLA: Thinking in Latent Spatio-Temporal Space for Vision-Language-Action in Autonomous Driving* (arXiv: 2603.01928v2), Yuechen Luo, Fang Li, Shaoqing Xu et al.
- Code: https://github.com/luo-yc17/LaST-VLA
- Related work: DriveWorld-VLA (arXiv: 2602.06521v1)
Architecture: Decoupled Thinking and Planning
The joint distribution is decomposed into: 1. Thinking: image + navigation instruction + ego state + history → physically grounded latent representation H 2. Planning: H → future trajectory points
A structured causal mask enforces:
Progressive Training
1. Phase I – Physics-Aware Alignment: λ_WM = λ_3D = 1.0 ≫ λ_action = 0.01; latent CoT strictly aligned with teacher representations; action tokens see only latent thought
2. Phase II – Latent-Grounded Planning: weights inverted (λ_action = 1.0); action tokens may also attend to raw images to refine driving policy
3. Phase III – GRPO Refinement: geometry/dynamics adapters frozen; reward = PDMS trajectory quality + format compliance + goal precision
Results
NAVSIM v1 (SOTA)
| Model | PDMS | NC | DAC | EP | |-------|------|-----|-----|-----| | LaST-VLA-8B | 91.3 | 98.7 | 97.9 | 86.7 | | LaST-VLA-2B | 91.1 | 98.6 | 97.7 | 86.4 | | DriveVLA-W0-7B | 86.1 | 96.5 | 94.0 | 80.0 | | Recogdrive-2B | 90.8 | 98.3 | 94.7 | 83.0 |
NAVSIM v2 (SOTA)
| Model | EPDMS | |-------|-------| | LaST-VLA-8B | 87.1 | | LaST-VLA-2B | 86.8 | | DriveVLA-W0-7B | 86.1 |
Spatial Reasoning (SURDS)
LaST-VLA-8B vs. InternVL3-8B: Yaw judgment +28.7% (70.16% vs 54.50%), pixel localization +25.8%, depth range +15.8%, left/right +9.1% (90.27%).
Dynamic Scene Understanding (NuDynamics)
LaST-VLA-8B reaches 81.19%, beating Qwen2.5-VL-72B (74.50%)—showing physical alignment matters more than scale.
Ablations
Key Takeaways
1. Physically grounded latent spatio-temporal reasoning is the next frontier for VLA models 2. Dual alignment (3D geometry + world model) injects physics more comprehensively than a single source 3. Progressive training (physics first, planning second, RL refinement last) is critical for stability 4. Alignment beats scale: the 2B model nearly matches the 8B model 5. Safety metrics (NC, DAC) show the largest gains, directly translating to real-world safety