OctoSense: Self-Supervised Multimodal Robot Perception with a 59-Hour Sensor Dataset

Forum topic · 小凯 · 2026-06-28

Summary

OctoSense is an open-source sensor platform and dataset for multimodal robot perception, combining stereo RGB cameras, event cameras, LiDAR, a thermal camera, an IMU, RTK-corrected GPS, and proprioceptive data (CAN bus signals from a car and joint angles from a quadruped robot). The accompanying dataset provides 59 hours of time-synchronized driving data across varied environments and times of day, including cases of severe sensor degradation. The authors introduce a late-fusion masked autoencoder for self-supervised multimodal learning that uses modality-specific tokenizers to handle differences in sensor representations, frequencies, latencies, and noise, and caches modality-specific tokens for incremental processing as new measurements arrive. The model computes representations in 6.68 ms on an NVIDIA 5090 and 112 ms on an Orin NX, outperforms image-only foundation models on optical flow, depth estimation, semantic segmentation, and egomotion prediction (translation, rotation, steering angle), and remains robust at night or when sensor data is degraded. Paper: arXiv 2606.27317.

Overview

Field: Computer Vision (CV) Authors: Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero, Pratik Chaudhari Published: 2026-06-25 arXiv: 2606.27317

Summary (Chinese post translated)

The authors present OctoSense, an open-source sensor platform equipped with stereo RGB cameras, an event camera, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected GPS, and proprioception sources (CAN bus data from a car and joint angles from a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data spanning different types of environments at different times of day, including situations with severely degraded sensor data.

The paper demonstrates multimodal self-supervised learning on real robot data, where sensors differ in representation, frequency, latency, and noise. The proposed method is a "late-fusion" masked autoencoder that:

1. Uses modality-specific tokenizers to adapt to the distinct spatiotemporal characteristics of each sensor. 2. Caches modality-specific tokens at inference time so new measurements can be processed incrementally as they arrive.

Key results

Fast inference: representations computed in 6.68 ms on an NVIDIA 5090 and 112 ms on an Orin NX.
Outperforms image-only foundation models on optical flow estimation, depth estimation, semantic segmentation, and egomotion (translation, rotation, and steering angle).
Robust predictions even at night or under severely degraded sensor conditions.

Original abstract (excerpt)

> We present OctoSense, an open-source sensor platform with stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles for a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors.

---

*Auto-collected on 2026-06-28*