DCVC-MB: Neural B-Frame Video Compression using State Space Models

Neural video codecs (NVCs) have historically struggled to effectively incorporate B-frames—a staple of traditional video compression standards like HEVC and VVC—due to the high computational complexity of bidirectional temporal fusion and the tendency of transformer-based architectures to scale quadratically with resolution. The authors seek to overcome these limitations by introducing DCVC-MB, a framework that leverages state-space models to enable efficient bidirectional prediction.

The authors propose DCVC-MB, which utilizes an IBP frame strategy to facilitate bidirectional temporal prediction. The core innovation is a Bidirectional Mamba Fusion architecture that replaces traditional transformer-based fusion modules. By using Mamba, the model achieves linear O(N) complexity, allowing it to process high-resolution video (e.g., 1080p) without exhausting VRAM. To further improve efficiency, the authors implement an adaptive latent skipping mechanism that selectively omits coding certain latents based on the frame's distribution, significantly reducing entropy coding time. Additionally, they introduce canonical positional embeddings to ensure the model generalizes across different resolutions during inference.

DCVC-MB demonstrates significant improvements in compression efficiency, measured by BD-rate reductions. It outperforms existing neural video codecs like DCVC-DC and DCVC-FM, as well as the traditional VTM-19.0 benchmark. Specifically, the method achieves BD-rate savings of up to 30.45% over VTM-19.0-LDP and 1.81% over VTM-19.0-RA (Inter-GoP=16). The ablation studies confirm that the Mamba-based fusion module is the primary driver of these gains, while inference-time strategies like open GoP and bidirectional coding provide additional, incremental improvements.

This work bridges a critical gap between traditional codecs and NVCs by demonstrating that B-frame coding can be implemented efficiently within a neural framework. By successfully applying state-space models to bidirectional temporal fusion, the authors provide a scalable path forward for high-resolution neural video compression that is competitive with, and in some cases superior to, established industry standards.