ResearchPod Summary
Streaming video understanding (SVU) requires models to process continuous video streams and answer asynchronous queries under strict GPU memory and latency constraints. A major challenge is the "delayed query" problem: decisive visual cues often appear long before a query is issued. Traditional methods either discard old information (sliding windows) or struggle with the memory-intensive task of retaining specific past tokens, which often leads to the loss of critical evidence as the delay increases.
ProtoKV introduces a two-tier memory architecture designed to maintain a constant memory footprint regardless of stream length.
ProtoKV demonstrates that summarizing far history into prototypes is more robust than retaining individual tokens. In long-delay regimes, ProtoKV achieves accuracy gains of up to 12.5 points over token-retention baselines. Because the prototype bank uses continuity-aware assignment—grouping tokens based on key similarity, spatial consistency, and recency—it effectively prevents decisive cues from being diluted or fragmented by subsequent updates. The performance gap between ProtoKV and standard baselines widens as the query delay increases, confirming the effectiveness of the summary-state approach for long-range temporal reasoning.
This research provides a practical, training-free solution for deploying Large Vision-Language Models (LVLMs) in real-time streaming environments. By decoupling memory capacity from stream length, ProtoKV enables more reliable long-term video reasoning without the unpredictable latency spikes associated with retrieval-based systems or the information loss inherent in simple sliding windows.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.