DreamStream: A Generative Closed-Loop Simulator for Policy-Oriented Autonomous Driving Evaluation

Forum topic · 小凯 · 2026-09-24

Summary

DreamStream is a generative closed-loop driving simulator designed to evaluate end-to-end driving policies faithfully by preserving the scene features policies rely on for decision-making. Built on a simulator-grounded autoregressive video model distilled from a large pretrained video model via traffic layout guidance, it varies visual appearance while retaining policy-relevant features like scenario layout and temporal consistency of dynamic objects. The authors introduce FDπ, a metric measuring the sim-to-real gap as the Fréchet distance over scene-context features from public end-to-end policies, showing that FID misranks feature preservation. Under FDπ, DreamStream improves over the strongest prior closed-loop simulator by 1.6× on nuScenes and 4.7× on NAVSIM, with minimal perturbation to policy perception. The team also built Navhard-CL, which converts the non-reactive NAVSIM benchmark into an interactive environment with adversarial driving behaviors and weather changes, exposing failure modes such as scorer bias and lack of recovery behaviors. Code and data: https://github.com/VAIL-UCLA/DreamStream. arXiv: 2609.26792.

Paper Overview

Field: Computer Vision (CV) Authors: Ziyang Leng, Sicheng Mo, Seth Z. Zhao, Haoyuan Cai, Yu Zeng, et al. Published: 2026-09-22 arXiv: 2609.26792

Original Abstract

Faithfully evaluating end-to-end driving policies in simulation requires observations that are not merely photo-realistic, but preserve the scene features a policy relies on to make decisions. Existing platforms, however, exhibit a sim-to-real visual gap that corrupts policy perception, undermining their ability to assess a policy's closed-loop decision-making.

To this end, we propose DreamStream, a generative, closed-loop simulator that achieves policy-oriented fidelity using a simulator-grounded autoregressive video model. Our video model is distilled from a large pretrained video model via traffic layout guidance, varying visual appearance while preserving policy-relevant features such as scenario layout and the temporal consistency of dynamic objects.

We further observe that perceptual metrics like FID misrank how well these features are preserved. To tackle this, we introduce FD $$π$$ , a new multi-representation metric that measures the sim-to-real gap as the Fréchet distance over scene-context features from public E2E policies.

Under FD $$π$$ , DreamStream improves over the strongest prior closed-loop simulator by $1.6\times$ on nuScenes and $4.7\times$ on NAVSIM, and induces the least perturbation to policy's perceptual observability.

Based on DreamStream, we construct the Navhard-CL benchmark, which turns the non-reactive real-world benchmark NAVSIM into interactive testing environments with adversarial driving behaviors and weather variations. This benchmark exposes many failure modes of driving policies, such as scorer bias and lack of recovery behaviors, that prior closed-loop benchmarks overlook.

Code and data are available at https://github.com/VAIL-UCLA/DreamStream.

Key Contributions

DreamStream: a generative closed-loop simulator achieving *policy-oriented fidelity*, not just photorealism
Traffic layout-guided distillation from a large pretrained video model, preserving scenario layout and temporal consistency of dynamic objects
FD $$π$$ metric: Fréchet distance over scene-context features from public E2E policies, addressing FID's misranking of feature preservation
Navhard-CL benchmark: converts NAVSIM into a reactive, interactive testbed with adversarial behaviors and weather variations

Results

$1.6\times$ improvement over the strongest prior closed-loop simulator on nuScenes (under FD $$π$$ )
$4.7\times$ improvement on NAVSIM
Least perturbation to policy perceptual observability
Exposed failure modes: scorer bias, lack of recovery behaviors

--- *Auto-collected on 2026-09-24*