Paper Overview
Field: Computer Vision Authors: Daniel Dauner, Valentin Charraut, Bastian Berle Published: 2025-05-07 arXiv: 2505.05127
Abstract
The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each dataset adopts different 2D and 3D modalities, such as cameras, lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple datasets.
We present 123D, an open-source framework that unifies such multi-modal driving data through a single API. To handle synchronization, we store each modality as an independent timestamped event stream, with no preset rate, supporting both synchronized and asynchronous access across arbitrary datasets. With 123D, we consolidate 8 real-world driving datasets spanning 3,300 hours and 90,000 km, plus a synthetic dataset with configurable collection scripts, and provide data analysis and visualization tooling.
Key Contributions
- Single unified API for multi-modal driving data across datasets
- Timestamped event-stream storage per modality, handling arbitrary synchronization schemes
- Integration of 8 real-world datasets: 3,300 hours, 90,000 km, plus one configurable synthetic dataset
- Systematic comparison of annotation statistics; evaluation of pose and calibration accuracy for each dataset
- Two demonstrated applications: cross-dataset 3D object detection transfer and reinforcement learning for planning
Original Abstract (Excerpt)
The pursuit of autonomous driving has produced one of the richest sensor data collections in all of robotics. However, its scale and diversity remain largely untapped. Each dataset adopts different 2D and 3D modalities, such as cameras, lidar, ego states, annotations, traffic lights, and HD maps, with different rates and synchronization schemes. They come in fragmented formats requiring complex dependencies that cannot natively coexist in the same development environment. Further, major inconsistencies in annotation conventions prevent training or measuring generalization across multiple datasets. We present 123D, an open-source framework that unifies such multi-modal driving data through a single API...
---
*Auto-collected on 2026-05-12.*