Sensor2Sensor: Turning Dashcam Video into Autonomous Driving Sensor Data

Forum topic · 小凯 · 2026-05-24

Summary

Sensor2Sensor (arXiv, cs.CV) by Jiahao Wang, Bo Sun, and Yijing Bai tackles a core bottleneck in autonomous driving: AV training requires expensive, structured, multi-modal sensor logs from test vehicles, while billions of hours of cheap, single-view dashcam footage worldwide are unusable by perception systems. The paper proposes translating in-the-wild monocular video into AV-log format—multi-view camera images plus LiDAR point clouds—without paired training data. The pipeline uses 4D Gaussian Splatting to reconstruct AV-log scenes and render dashcam-style videos from new viewpoints, creating synthetic paired data with perfect ground truth. A diffusion model is then trained to conditionally generate multi-view images and synchronized LiDAR point clouds from dashcam-style input, with cross-modal alignment losses ensuring consistency. Evaluations cover fidelity (FID), geometric, temporal, and cross-modal consistency, and downstream task performance. The approach unlocks long-tail scenarios—sandstorms, wildlife crossings, icy roads—captured only by ordinary cameras, potentially democratizing autonomous driving data away from a few deep-pocketed fleet operators.

Sensor2Sensor: Turning Dashcam Video into Autonomous Driving Sensor Data

*Adapted from a Chinese forum post on zhichai.net discussing the paper "Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving" by Jiahao Wang, Bo Sun, and Yijing Bai (arXiv, cs.CV).*

Key points

The gap between two data worlds. Autonomous driving systems (ADS) train on expensive AV logs: structured, multi-modal, time-synchronized data from 8–12 camera arrays, LiDAR point clouds, radar, and high-precision GPS/IMU on multi-million-dollar test fleets. Meanwhile, hundreds of millions of dashcams worldwide generate massive single-view, unstructured video that ADS cannot directly consume. AV-log data is scarce, geographically limited, and almost never covers long-tail scenarios (sandstorms, wildlife, floods, extreme accidents).
The core problem: no parallel data. Translating monocular dashcam video into multi-view images + LiDAR point clouds is a cross-modal, cross-embodiment translation task with no paired corpus—a test car and a taxi cannot practically record the same scene from identical viewpoints.
Solution step 1: 4D Gaussian Splatting (4DGS) bootstrapping. The authors reconstruct dynamic 3D scenes from real AV logs using 4DGS, then render them from virtual dashcam-like viewpoints (monocular, low-mounted, wide-angle). This produces realistic "dashcam-style" videos with perfect ground truth—the synthetic paired data needed for training, scalable to arbitrary camera poses.
Solution step 2: diffusion-based translation. A diffusion model learns to map dashcam-style input to AV-log output, with:
a condition encoder injecting dashcam features into the denoising process;
multi-modal output heads jointly generating multi-view images and LiDAR point clouds;
cross-modal alignment losses ensuring camera images and point clouds stay consistent in geometry and time.
Unlocking long-tail data. The paper demonstrates converting challenging in-the-wild internet footage—desert sandstorms, a wild elephant crossing a road, vehicles skidding on ice, rare multi-car pileups—into multi-view + LiDAR format that autonomous driving systems can train on.
Evaluation dimensions. The authors report comprehensive quantitative and qualitative assessment covering fidelity (e.g., FID), geometric consistency of generated point clouds, temporal consistency across frames, cross-modal consistency, and downstream task performance (detection, segmentation, depth estimation trained on generated data).

Why it matters

AV data is effectively monopolized by giants (Waymo, Tesla, Cruise, Baidu Apollo, XPeng) that can afford large test fleets. Smaller teams rely on public datasets like nuScenes, KITTI, and Waymo Open Dataset—tiny by comparison. Sensor2Sensor suggests a path to data democratization: billions of hours of existing public/semi-public dashcam footage covering every road, weather condition, and driving culture could be converted into usable training fuel, addressing long-tail risk that no test fleet will ever deliberately collect.

The philosophical caveat remains: LiDAR point clouds "imagined" by a diffusion model are not pixel-perfect reality. The paper's answer is rigorous evaluation—generated data must be statistically faithful enough to serve as a valid supplement to real data, not a replacement.

Paper: Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving — Jiahao Wang, Bo Sun, Yijing Bai (arXiv, cs.CV)

Sensor2Sensor: Turning Dashcam Video into Autonomous Driving Sensor Data

Summary

Sensor2Sensor: Turning Dashcam Video into Autonomous Driving Sensor Data

Key points

Why it matters

Tags