Towards Fast and Effective Long Video Understanding of Multimodal Large Language Models via Adaptive Quasi-Gaussian Sampling

Long video understanding is computationally expensive for Multimodal Large Language Models (MLLMs) because processing every frame leads to excessive memory and latency. While keyframe selection is a common mitigation, existing methods often struggle with flexibility—performing well on local 'needle-in-a-haystack' tasks but failing on global understanding—and are sensitive to noise in frame-query similarity scores.

The authors propose AdaQ, a training-free framework that treats frame selection as a 'Quasi-Gaussian Sampling' problem. Inspired by the 3-sigma rule of Gaussian distributions, AdaQ defines an effective sampling interval where frames outside this range are treated as noise. Crucially, the method uses the variance of frame-query similarity scores to dynamically adjust the temperature of the distribution. This allows the model to automatically shrink the sampling interval for precise local queries and expand it for broader global queries.

AdaQ was evaluated across four MLLMs (including Qwen3-VL and LLaVA-OneVision) and three embedding models (CLIP, LongCLIP, BLIP). The results show that:

• AdaQ consistently outperforms both default uniform sampling and state-of-the-art keyframe selection methods across multiple benchmarks like Video-MME and LVBench.
• The method is highly efficient, requiring only one hyper-parameter (gamma) to be set.
• By adaptively adjusting the 3-sigma interval, AdaQ helps models like Qwen3-VL-8B achieve significant performance gains, in some cases outperforming models like GPT-4o by over 15% on average while using only 64 frames.

This research provides a robust, plug-and-play solution for making long-video MLLMs more efficient without requiring expensive retraining or fine-tuning. By demonstrating that similarity variance is a reliable indicator for task-specific sampling, the authors offer a simple yet effective way to bridge the gap between local and global video understanding, making advanced MLLMs more practical for resource-constrained environments.