Paper Overview
Research Area: CV Authors: Jiahao Wang, Bo Sun, Yijing Bai arXiv: 2505.14490
Summary
Robust training and validation of autonomous driving systems (ADS) requires large-scale, diverse datasets. Proprietary data collected by AV fleets is high fidelity but limited in scale, sensor configuration diversity, and coverage of geographic and long-tail behaviors. In contrast, wild data from sources such as dashcams offers enormous scale and diversity, capturing critical long-tail scenarios and novel environments. However, this unstructured wild video data is incompatible with ADS that expect structured, multimodal sensor inputs for validation and training.
To bridge this data gap, the authors propose Sensor2Sensor, a novel generative modeling paradigm that converts wild monocular dashcam videos into high-fidelity multimodal sensor suites (AV logs), including multi-view camera images and LiDAR point clouds.
Key Challenge and Approach
- Core challenge: lack of paired training data (no real dashcam-to-AV-log pairs exist).
- Solution: convert real AV logs into dashcam-style videos via 4D Gaussian Splatting (4DGS) reconstruction and novel view rendering, creating synthetic pairs for training.
- Sensor2Sensor then uses a diffusion-based architecture to perform the generative conversion.
- Comprehensive quantitative evaluation of the fidelity and realism of the generated sensor data.
- Demonstrated practical value by converting challenging wild internet and dashcam clips into realistic multimodal data formats.
- Unlocks massive external data sources for autonomous driving development.
Evaluation and Impact
*Auto-collected on 2026-05-25*