OctoSense: Self-Supervised Learning for Multimodal Robot Perception

Research Overview

Robotic perception often fails in real-world conditions because individual sensors—such as cameras, LiDAR, or IMUs—have distinct failure modes, noise profiles, and environmental sensitivities. While self-supervised learning (SSL) has advanced visual foundation models, these models rarely account for the full suite of sensors available on modern robots. The authors introduce OctoSense, a comprehensive open-source platform and dataset designed to bridge this gap by providing time-synchronized data across diverse modalities.

The OctoSense Dataset

The OctoSense dataset contains 59 hours of synchronized data collected from a vehicle equipped with stereo RGB and event cameras, thermal imaging, LiDAR, IMU, RTK-corrected GPS, and proprioceptive sensors. The data covers urban, suburban, and rural environments across various times of day and weather conditions, including scenarios with significant sensor degradation. This dataset is uniquely suited for multi-modal research due to its scale and the inclusion of high-frequency event camera data alongside traditional sensors.

Late-Fusion Masked Autoencoder

To process this heterogeneous data, the authors propose a "late-fusion" masked autoencoder (MAE). The architecture employs modality-specific tokenizers that account for the unique spatiotemporal characteristics of each sensor—such as the high-frequency stream of event cameras versus the sparse point clouds of LiDAR. By caching modality-specific tokens, the model achieves high inference speeds, making it suitable for real-time robotic applications. The model is trained to reconstruct masked inputs, forcing it to learn a robust, fused representation that integrates information across different modalities.

Performance and Impact

The authors demonstrate that their multi-modal approach significantly outperforms existing image-only SSL models on downstream tasks, including optical flow, depth estimation, semantic segmentation, and ego-motion prediction. The performance gains are most pronounced in challenging environments, such as nighttime driving or when sensors are degraded by sunflares or packet loss. This work provides a scalable framework for building perception systems that are resilient to the environmental challenges that typically keep field robots indoors.