Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding

As multimodal large language models (MLLMs) are applied to longer video streams, they face a critical bottleneck: the linear growth of visual tokens leads to prohibitive computational and memory costs. Existing compression methods often rely on global video access or pre-defined queries, making them unsuitable for real-time streaming scenarios where future content is unpredictable. This paper asks: can we construct a dynamic, fixed-budget visual memory bank that preserves essential semantics under strict causal constraints?

The authors propose CausalMem, a training-free method that maintains a fixed-size memory bank for streaming video. The core innovation is an online semantic basis—a compact set of vectors that captures the principal semantics of the observed video stream. As new frames arrive, CausalMem projects visual tokens onto this basis to estimate their redundancy. Tokens with high redundancy (those well-represented by the current basis) are discarded, while informative tokens are retained. To ensure the memory bank remains relevant to the evolving stream, the system balances this redundancy score with a temporal recency prior, ensuring that the memory bank effectively maximizes information gain within a strictly bounded storage footprint.

CausalMem demonstrates significant improvements in both streaming and offline video understanding benchmarks. When applied to LLaVA-OneVision and Qwen2.5-VL, the method consistently outperforms existing streaming compression techniques, achieving average accuracy gains of +3.2% on streaming benchmarks and +3.0% on offline benchmarks. Notably, CausalMem enables the processing of hour-long videos using only a 12k token budget, resulting in a 20x compression ratio and a storage footprint of approximately 82 MB, all while maintaining superior semantic preservation compared to existing methods.