Event-RGB Adaptive Tracking for Nighttime Highway Perception

Conventional Intelligent Transportation Systems (ITS) rely heavily on RGB cameras, which suffer from catastrophic performance degradation in nighttime highway environments due to motion blur, underexposure, and high vehicle speeds. While event cameras offer high temporal resolution to mitigate these issues, they are blind to stationary objects. This paper addresses the challenge of creating a robust, multi-modal tracking system that can dynamically leverage the complementary strengths of RGB and event sensors without relying on rigid, hard-coded prioritization.

The authors propose the Joint Event-RGB Adaptive Tracking (JEAT) framework. Unlike traditional trackers that prioritize one sensor over the other, JEAT merges detections from both modalities into a unified data association optimization problem. The core of the framework is an Adaptive Extended Kalman Filter (AEKF) that uses the Normalized Innovation Squared (NIS) statistic to estimate measurement noise in real-time. By continuously updating the noise covariance for each sensor, the system automatically shifts its trust: it favors event streams during high-speed motion or low-light conditions and relies on RGB frames when objects are static or lighting is sufficient. To support this research, the authors also introduce SEHN, a large-scale synthetic dataset generated via the CARLA simulator, which provides synchronized RGB and event data across diverse nighttime highway scenarios.

Findings

The JEAT framework effectively resolves cross-modal failure modes. By treating the fusion as a joint optimization problem, the system ensures that the most reliable sensor at any given moment dictates the tracking state. The adaptive noise estimation allows the tracker to remain stable even as environmental conditions change rapidly. The authors demonstrate that this approach significantly improves tracking reliability compared to fixed-priority fusion methods, particularly in scenarios where vehicles move at high velocities in the absence of artificial lighting.