Gaussian Mixture Modeling for Event-Aware Visual Allocation in Long Video Understanding

Large Vision-Language Models (LVLMs) struggle with long video understanding due to the high computational cost of processing many frames. Existing methods often sample frames uniformly or treat them as isolated units, which leads to redundant visual tokens and the loss of high-level semantic structure. This paper asks: can we improve the efficiency and accuracy of long video understanding by explicitly modeling the temporal event structure of a video to guide visual budget allocation?

The authors propose GMM-EVA (Gaussian Mixture Modeling for Event-Aware Visual Allocation), a training-free, plug-and-play method. The process involves three main stages:

1. Relevance Modeling: Using an off-the-shelf cross-modal embedding model (like CLIP), the system computes frame-wise relevance scores relative to a textual query.
2. Latent Event Discovery: The system fits a Gaussian Mixture Model (GMM) to these scores using the Expectation-Maximization (EM) algorithm. Each Gaussian component represents a latent semantic event, capturing the continuous temporal span of important video segments.
3. Event-Aware Allocation: The system assigns each frame to its most probable event. Within each event, the most relevant frame is designated as a high-resolution primary keyframe (a semantic anchor), while remaining frames are treated as low-resolution secondary keyframes to maintain temporal context.

GMM-EVA significantly outperforms uniform sampling across multiple long-video benchmarks. By concentrating high-resolution visual tokens on primary keyframes and reducing the budget for secondary frames, the method achieves performance comparable to or better than state-of-the-art selection methods while consuming only about half the visual token budget. The framework is robust, generalizing well across different embedding backbones and downstream LVLMs.