Hongyu Qu, Guangming Yao, Ling Xing, Xiaobin Hu, Rongxing Ding, Guibin Zhang, Fan Zhang, Yi Yuan, Xiangbo Shu, Shuicheng Yan
5 min
Abstract
Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
Sam: Exactly. And that's what lets the latent state become task-relevant rather than just historically complete. The entropy signal forces the LMTs to prioritize information that actually reduces uncertainty about what's being asked — which is a much tighter objective than "store everything and retrieve the most similar frames."
Alex: What does that cost in practice?
Sam: The main cost is latency to first token. You're running an optimization loop before generation starts, so that initial pause is longer. But decoding then becomes substantially faster, because the model is working from a compact, already-integrated state rather than attending over a growing context window. For real-time applications, that's actually a favorable exchange: a fixed upfront cost versus unbounded per-token overhead as the stream lengthens.
Alex: And because the backbone is frozen throughout, you avoid catastrophic forgetting — the model's pretrained capabilities stay intact while the memory adapts.
Sam: That's the other structural advantage. The memory evolution is entirely in the LMT space, so you're not touching the weights that give the model its general reasoning ability. It also means you don't need labeled data for the adaptation — the entropy signal is self-supervised.
Alex: Where would a careful referee push back?
Sam: A few places. The test-time optimization adds a latency cost that the paper acknowledges but doesn't fully characterize across different stream lengths and query types, so it's not clear how that scales in the worst case. The Jenks clustering is a reasonable choice for temporal segmentation, but it's not obviously optimal, and the ablation on that design choice would be worth scrutinizing. And the entropy-based reward assumes that reducing model uncertainty is a good proxy for task relevance — which holds in many settings but can break down when the model is confidently wrong, a known failure mode for entropy-based objectives.
Alex: So the core claim — that internalizing history into an evolving latent state outperforms static retrieval — rests on both the architectural design and the quality of that entropy signal holding up.
Sam: Those are the load-bearing pieces. The hierarchical memory and the LMT optimization are what make "retrieve-and-internalize" work in practice, and the entropy reward is what gives the system a principled stopping criterion. If either degrades — say, entropy becomes a poor signal for a particular domain — the whole framework needs revisiting. But as a proof of concept that test-time optimization can substitute for ever-growing context windows in streaming video, LatentStream makes a coherent case.
Alex: A compact latent state that refines itself at inference, guided by its own uncertainty. That's a clean design principle, and one worth watching as streaming video tasks get more demanding. Thanks for listening to ResearchPod.