Sensor2Sensor: Turning Dashcam Footage into LiDAR and Multi-Camera Data for Autonomous Driving

> "Data is the new oil, but oil itself does not distinguish between nobles and commoners. What matters is who owns the refinery." > — adapted from Clive Humby

The Two Worlds of Driving Data

Hundreds of millions of dashcams worldwide record monocular video every second — rich, wild, unstructured footage of cities, weather, and traffic. Autonomous driving systems (ADS), however, cannot directly consume this data. ADS development relies on AV logs: structured, multimodal, time-synchronized sensor records from test vehicles equipped with:

• 8–12 cameras (narrow/wide FOV, side, fisheye surround view)
• LiDAR producing centimeter-accurate 3D point clouds with (x, y, z) coordinates and reflectance intensity
• Millimeter-wave radar for rain/fog-robust speed and range measurement
• High-precision GPS/IMU, wheel encoders, steering angle sensors

This first world of data is noble, expensive, controlled, and scarce: a test vehicle plus operations costs millions of dollars, only a few thousand test vehicles exist globally, and long-tail scenarios (dust storms, wild animals, flooded streets) are essentially never collected.

The second world — billions of hours of dashcam video — is cheap, ubiquitous, and endless, but unstructured, single-modality, low-precision, and viewpoint-limited.

The paper "Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving" by Jiahao Wang, Bo Sun, and Yijing Bai asks: *what if we could translate the second world into the format of the first?*

Translation Without a Dictionary

This is a cross-modal, cross-embodiment translation task with a catch: there is no paired corpus. You cannot have a test vehicle and a taxi drive the same intersection at identical speed and angle recording simultaneously. It is like training a translator with no bilingual dictionary — only monolingual texts in each language.

Step 1: 4D Gaussian Splatting Used in Reverse

4D Gaussian Splatting (4DGS) represents a dynamic 3D scene (space + time) as millions of tiny anisotropic 3D Gaussians, each with position, color, opacity, size, and time-varying parameters. Rendered from any viewpoint, these overlapping "bubbles" composite into photorealistic images — differentiable, efficient, and geometrically precise.

The standard forward use: multi-view photos + known camera poses → 4DGS reconstruction → novel-view rendering.

Sensor2Sensor's clever inversion:

1. Take real AV logs (multi-view images + LiDAR point clouds + precise poses) 2. Reconstruct the scene with 4DGS 3. Render from new viewpoints mimicking a dashcam (monocular, low-mounted, fisheye) 4. Produce videos that *look* like dashcam footage — with perfect ground truth

This yields synthetic paired data: the same scene in both dashcam-style input and full AV-log output (multi-view + LiDAR). Like a bilingual speaker writing the same sentence in two languages, it is enough to train the translation model — and it scales to arbitrary virtual camera poses.

Step 2: Diffusion Models as the Translator

With paired data in hand, Sensor2Sensor trains a diffusion model — the engine behind Stable Diffusion and DALL-E — to perform the translation. A diffusion model learns to reverse gradual noising, recovering structure step by step. Here:

• Input: a low-information monocular dashcam frame (single viewpoint, no depth, lens distortion, compression artifacts)
• Output: the corresponding multi-view camera images + LiDAR point cloud in AV-log format

Key architectural elements described in the post:

• Conditional encoder: injects dashcam features into the denoising process, keeping outputs semantically consistent with the input scene
• Multi-modal output heads: jointly generate multi-view images and LiDAR point clouds with spatiotemporal consistency
• Cross-modal alignment loss: camera-visible objects must appear in the point cloud and vice versa

Unlocking Long-Tail Data

The authors demonstrate conversion of challenging in-the-wild footage into realistic multimodal data: sandstorms on desert highways, wild elephants crossing roads, vehicles skidding on ice, multi-car pileups. Test fleets would never (and could never safely) collect these — yet they are precisely the long-tail risk scenarios where perception models fail most. Converted videos become:

• 360° multi-view camera imagery
• LiDAR point clouds with depth and reflectance
• Time-synchronized multimodal sensor logs

Is Generated Data "Real"?

The post raises the philosophical question of whether diffusion-imagined LiDAR counts as real. The answer lies in rigorous evaluation across:

• Fidelity — FID and human assessment of image realism
• Geometric consistency — depth accuracy and alignment of generated point clouds against the reconstructed scene
• Temporal consistency — no flicker or jumps across frames
• Cross-modal consistency — image/point-cloud agreement
• Downstream task performance — models trained on generated data improving on real data for detection, segmentation, and depth estimation

The result: statistically "real enough" to serve as an effective supplement to collected data.

Democratizing Driving Data

Today, only giants (Waymo, Tesla, Cruise, Baidu Apollo, XPeng) can afford massive AV-log accumulation, producing data concentration, geographic limitation, and long-tail poverty. Sensor2Sensor offers an alternative: billions of hours of public or semi-public dashcam footage already on the internet, converted rather than stolen — a format translation that lowers the data barrier for startups, researchers, and labs.

The vision: every vehicle as both user and contributor, every dashcam a node in a vast decentralized training network — where a taxi's dashcam becomes, in effect, a LiDAR for autonomous driving AI.

> "True innovation is not inventing new eyes, but letting old eyes see new things."

---

Paper info:

• Title: Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving
• Authors: Jiahao Wang, Bo Sun, Yijing Bai
• arXiv category: cs.CV