ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression

Learned video compression relies on accurate temporal modeling to remove redundancy between frames. However, standard codecs infer motion solely from RGB frames, which are discrete samples of a continuous scene. This makes them vulnerable to fast motion, blur, occlusion, and low-light conditions. The authors investigate whether event cameras—which capture asynchronous, high-temporal-resolution intensity changes—can provide the missing dynamic information to improve motion estimation and, consequently, compression efficiency.

ENCORE (Event-Assisted Complementary Motion Refinement) is introduced as an auxiliary-modality framework. It integrates into existing RGB codecs (using HyTIP as a backbone) without changing the decoder or the final reconstruction target. The framework consists of three main modules:

• Complementary Motion Representation (CMR): Decomposes aligned RGB and event features into common motion, RGB-specific, and event-specific representations.
• Spatial Energy and Redundancy-Informed Calibration (SERIC): Evaluates event-specific features to identify novel, active motion cues while suppressing noise or redundant information already captured by RGB.
• Energy-Aware Routing (EAR): Determines the spatial locations and intensity with which the event-derived flow correction is applied to the initial RGB-based motion estimate.

Experiments across three datasets (BS-ERGB, HQ-EVFI, and CED) show that ENCORE consistently improves rate-distortion performance. On the BS-ERGB dataset, the framework achieved up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings compared to the RGB-only baseline. The authors demonstrate that these gains are robust across different Group of Pictures (GOP) lengths and capture conditions, with a modest computational overhead of approximately 7.3% increase in GFLOPs per P-frame.