Paper Overview
Field: Computer Vision Authors: Jiahao Wang, Bo Sun, Yijing Bai Published: 2025-05-23 arXiv: 2505.17379
Summary
Robust training and validation of autonomous driving systems (ADS) require large-scale, diverse datasets. Proprietary data collected by autonomous driving fleets, while high in fidelity, is limited in scale, sensor configuration diversity, and geographic and long-tail behavior coverage. In contrast, in-the-wild data from sources such as dashcams offers tremendous scale and diversity, capturing critical long-tail scenarios and novel environments. However, this unstructured wild video data is incompatible with the structured multimodal sensor input that ADS expect for validation and training.
To bridge this data gap, the authors propose Sensor2Sensor, a novel generative modeling paradigm that converts in-the-wild monocular dashcam video into high-fidelity multimodal sensor suites (AV logs), including multi-view camera images and LiDAR point clouds.
The core challenge is the lack of paired training data. The authors address this by converting real AV logs into dashcam-style videos via 4D Gaussian Splatting (4DGS) reconstruction and novel-view rendering. Sensor2Sensor then uses a diffusion architecture to perform the generative conversion.
Contributions
- A generative paradigm (Sensor2Sensor) that transforms monocular dashcam video into full multimodal AV sensor suites (multi-view cameras + LiDAR).
- A pseudo-pairing strategy using 4DGS-based reconstruction and novel-view rendering of real AV logs to overcome the absence of paired data.
- Comprehensive quantitative evaluation of the fidelity and realism of the generated sensor data.
- Demonstrated practical utility: converting challenging in-the-wild internet and dashcam footage into realistic multimodal data formats, unlocking large external data sources for autonomous driving development.