123D: An Open-Source Framework Unifying Multi-Modal Autonomous Driving Data at Scale

Forum topic · 小凯 · 2026-05-12

Summary

This forum post introduces 123D, an open-source framework (arXiv:2505.05127) that unifies multi-modal autonomous driving data through a single API. Autonomous driving datasets are fragmented: each uses different 2D/3D modalities—cameras, lidar, ego states, annotations, traffic lights, and HD maps—with varying rates, synchronization schemes, and incompatible formats. Inconsistent annotation conventions further prevent cross-dataset training and generalization evaluation. 123D addresses synchronization by storing every modality as independent timestamped event streams without fixed rates, enabling both synchronous and asynchronous access across arbitrary datasets. Using the framework, the authors integrate 8 real-world driving datasets covering 3300 hours and 90,000 kilometers, plus a synthetic dataset with configurable capture scripts, along with analysis and visualization tools. They systematically compare annotation statistics and assess pose and calibration accuracy per dataset. Two demonstrated applications include cross-dataset 3D object detection transfer and reinforcement learning for planning.

Paper Overview

Research area: Computer Vision Authors: Daniel Dauner, Valentin Charraut, Bastian Berle Published: 2025-05-07 arXiv: 2505.05127

Abstract (Translated)

The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each dataset adopts different 2D and 3D modalities, such as cameras, lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple datasets.

We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, each modality is stored as an independent timestamped event stream without preset rates, supporting synchronous or asynchronous access across arbitrary datasets. Using 123D, the authors integrate 8 real-world driving datasets covering 3300 hours and 90,000 km, plus a synthetic dataset with configurable capture scripts, along with data analysis and visualization tools. They systematically compare annotation statistics and evaluate pose and calibration accuracy for each dataset. Finally, they demonstrate two applications enabled by 123D: cross-dataset 3D object detection transfer and reinforcement learning for planning, and outline directions for future work.

Highlights

Single unified API for heterogeneous driving datasets
Timestamped event-stream storage removes rigid synchronization constraints
8 real-world datasets: 3300 hours, 90,000 km, plus synthetic data
Evaluation of annotation, pose, and calibration consistency across datasets
Demonstrated use cases: cross-dataset 3D detection transfer and RL-based planning