OctoSense: Self-Supervised Multimodal Learning for Robot Perception (arXiv 2606.27317)

Forum topic · 小凯 · 2026-06-28

Summary

OctoSense is an open-source multimodal sensor platform and dataset for robot perception research, presented by researchers including Anthony Bisulco, Kostas Daniilidis, and Pratik Chaudhari (arXiv 2606.27317). The platform integrates stereo RGB cameras, an event camera, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected GPS, and proprioception from a car's CAN bus and a quadruped robot's joint angles. The accompanying OctoSense dataset provides 59 hours of time-synchronized driving data across varied environments and times of day, including cases of severely degraded sensors. The proposed method is a late-fusion masked autoencoder that uses modality-specific tokenizers to handle differing spatiotemporal characteristics, frequencies, latencies, and noise across sensors, and caches modality-specific tokens at inference time to process new measurements as they arrive. It computes representations in 6.68 ms on an NVIDIA RTX 5090 and 112 ms on an Orin NX, outperforms image-only foundation models on optical flow, depth estimation, semantic segmentation, and egomotion (translation, rotation, steering angle), and remains robust at night or under sensor degradation.

Paper Overview

Field: Computer Vision Authors: Anthony Bisulco, Jeremy Wang, Kostas Daniilidis, Randall Balestriero, Pratik Chaudhari arXiv: 2606.27317

Summary

OctoSense is an open-source sensor platform featuring stereo RGB and event cameras, LiDAR, a thermal camera, an inertial measurement unit, RTK-corrected global positioning system, and proprioception (CAN bus data from a car, and joint angles from a quadruped robot). The eponymous OctoSense dataset contains 59 hours of time-synchronized driving data across different types of environments at different times of the day, including situations with highly degraded sensors.

The work demonstrates multi-modal self-supervised learning using real-world robotics data, where sensors have different representations, frequencies, latencies, and noise. The approach is a "late-fusion" masked autoencoder that:

1. Uses modality-specific tokenizers to account for the different spatiotemporal characteristics of the sensors. 2. Caches modality-specific tokens at inference time so new measurements can be processed as they arrive.

Key Results

Fast inference: representations computed in 6.68 ms on an NVIDIA 5090 and 112 ms on an Orin NX.
Strong performance: outperforms existing image-only foundation models on optical flow estimation, depth estimation, semantic segmentation, and autonomous egomotion (translation, rotation, and steering angle).
Robustness: maintains reliable predictions at night or when sensor data is severely degraded.

OctoSense: Self-Supervised Multimodal Learning for Robot Perception (arXiv 2606.27317)

Summary

Paper Overview

Summary

Key Results

Links

Tags