Streaming video understanding requires multimodal large language models (MLLMs) to process continuous visual inputs and respond to user queries under strict causality and bounded memory. Existing approaches typically compress historical observations into an external memory bank and retrieve query-relevant evidence as additional visual context. Though effective, this store-and-retrieve paradigm keeps historical evidence as external visual context, preventing it from being internalized into a compact, evolving latent memory that can continuously guide streaming reasoning. To bridge this gap, we introduce LatentStream, a progressive latent working memory framework that shifts streaming memory from store-and-retrieve to retrieve-and-internalize. Specifically, LatentStream comprises three coordinated components. First, Query-agnostic Hierarchical Streaming Memory organizes visual history into short-, mid-, and long-term levels under a fixed memory budget through Jenks-guided adaptive consolidation. Once a query arrives, Hierarchical Latent Memory Evolution equips groups of latent memory tokens with progressively expanding memory receptive fields, enabling them to iteratively retrieve historical evidence from their corresponding scopes and internalize it into a compact, fixed-length latent memory. Finally, Progressive Confidence-guided Latent Memory Optimization constructs a hierarchical progression reward from group-wise predictive entropy and jointly refines the latent memory tokens and retrieved evidence, encouraging increasingly confident streaming reasoning. Extensive experiments demonstrate that LatentStream achieves new state-of-the-art results on existing online and offline video benchmarks.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at LatentStream — a new architecture for how AI models handle long-term video streams. The core problem is something the authors call "store-and-retrieve": the model keeps a bank of historical frames and pulls from it when a query arrives. History is treated as static context — raw data sitting in a buffer that the model never actually integrates. So as streams get longer, you're either drowning in context or losing information you've already seen.
Alex: So the buffer grows, retrieval gets expensive, and the model still isn't reasoning over history — it's just pattern-matching against stored frames?
Sam: Right. And the authors argue that's the wrong abstraction entirely. Their alternative is what they call "retrieve-and-internalize." Instead of accumulating raw frames, the system distills history into a compact, evolving latent state. The analogy they use is a student who actively summarizes material as they go, rather than highlighting every page and hoping to find it later.
Alex: That's a meaningful distinction. But how do you do that without retraining the backbone every time new frames arrive?
Sam: That's the technical crux. They introduce Latent Memory Tokens — LMTs — optimizable embedding vectors that serve as the model's working memory. The backbone stays frozen. When a query arrives, the LMTs retrieve evidence from a hierarchical memory bank, and then the tokens themselves get updated through a test-time optimization loop.
Alex: What's the hierarchy doing exactly?
Sam: They use a clustering algorithm called Jenks Natural Breaks to organize visual information across different temporal scales. Older, consolidated data lives at coarser levels; recent frames are represented more granularly. The LMTs are partitioned into groups with expanding receptive fields, so different token groups pull from different temporal resolutions. It keeps the memory footprint under a fixed budget without simply discarding older content.
Alex: And the optimization — how does the system know which direction to update the LMTs? What's the reward signal?
Sam: Predictive entropy. If the model is uncertain about its next prediction, entropy is high. The system treats that uncertainty as a signal that the current latent state isn't doing its job, and it backpropagates through the LMTs to reduce it. So the tokens are literally optimizing themselves at inference time to become more confident as they integrate history. The backbone parameters never move — this is test-time optimization, not fine-tuning.
Alex: That's a notable departure from retrieval-augmented generation, where the retrieval step is essentially fixed once the index is built. Here the memory representation itself is a learnable variable at inference.
Sam: Exactly. And that's what lets the latent state become task-relevant rather than just historically complete. The entropy signal forces the LMTs to prioritize information that actually reduces uncertainty about what's being asked — which is a much tighter objective than "store everything and retrieve the most similar frames."
Alex: What does that cost in practice?
Sam: The main cost is latency to first token. You're running an optimization loop before generation starts, so that initial pause is longer. But decoding then becomes substantially faster, because the model is working from a compact, already-integrated state rather than attending over a growing context window. For real-time applications, that's actually a favorable exchange: a fixed upfront cost versus unbounded per-token overhead as the stream lengthens.
Alex: And because the backbone is frozen throughout, you avoid catastrophic forgetting — the model's pretrained capabilities stay intact while the memory adapts.
Sam: That's the other structural advantage. The memory evolution is entirely in the LMT space, so you're not touching the weights that give the model its general reasoning ability. It also means you don't need labeled data for the adaptation — the entropy signal is self-supervised.
Alex: Where would a careful referee push back?
Sam: A few places. The test-time optimization adds a latency cost that the paper acknowledges but doesn't fully characterize across different stream lengths and query types, so it's not clear how that scales in the worst case. The Jenks clustering is a reasonable choice for temporal segmentation, but it's not obviously optimal, and the ablation on that design choice would be worth scrutinizing. And the entropy-based reward assumes that reducing model uncertainty is a good proxy for task relevance — which holds in many settings but can break down when the model is confidently wrong, a known failure mode for entropy-based objectives.
Alex: So the core claim — that internalizing history into an evolving latent state outperforms static retrieval — rests on both the architectural design and the quality of that entropy signal holding up.
Sam: Those are the load-bearing pieces. The hierarchical memory and the LMT optimization are what make "retrieve-and-internalize" work in practice, and the entropy reward is what gives the system a principled stopping criterion. If either degrades — say, entropy becomes a poor signal for a particular domain — the whole framework needs revisiting. But as a proof of concept that test-time optimization can substitute for ever-growing context windows in streaming video, LatentStream makes a coherent case.
Alex: A compact latent state that refines itself at inference, guided by its own uncertainty. That's a clean design principle, and one worth watching as streaming video tasks get more demanding. Thanks for listening to ResearchPod.