Killian Steunou, Anas Filali Razzouki, Khalil Guetari, Mounîm A. El-Yacoubi, Yannis Tevissen
4 min
Abstract
Video-language models can process only a limited number of frames, making frame selection a key bottleneck for efficient video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling has recently emerged as a promising approach for selecting the most informative frames from a video; however, existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. We find that, overall, on ActivityNet Captions and MSR-VTT, our method outperforms state-of-the-art methods across all evaluated downstream vision language models, especially when only one or two frames are selected for captioning, obtaining the best CIDEr for most frame budgets. On ActivityNet Captions, PEEK is particularly strong, winning 14 out of 16 configurations. Zero-shot evaluation on MSR-VTT shows that our model transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only $5.2\%$ to the captioning time, compared with $65.4\%$ for CSTA and $211.9\%$ for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek.
Sam: Precisely. And that's why it's so much faster in practice. Other adaptive methods require running a heavy AI model over the entire video every single time you want to process something. PEEK's student model adds only around five percent to the total processing time. It's lightweight, but it's still making informed choices rather than blind ones.
Alex: That said — if the system is being that selective, isn't there a risk it just keeps picking the same kind of frame over and over? Like, always grabbing the most visually dramatic moment at the start of a video?
Sam: That's a genuine concern, and the researchers addressed it directly. They use a technique called stratified argmax. The name sounds technical, but the idea is straightforward. Imagine dividing a video into equal chapters — say, ten sections of equal length. The model is required to pick exactly one frame from each chapter, choosing whichever is the most informative within that section. It can't just grab ten frames from the opening minute and ignore the rest.
Alex: So it's a built-in rule that forces the system to spread its attention across the whole video.
Sam: Exactly. It balances picking the best available content in any given moment with making sure the whole timeline is represented. You get quality and coverage, rather than having to choose between them.
Alex: And does it actually perform better than just sampling uniformly?
Sam: The results suggest it does, particularly when the system is working with a very tight frame budget — meaning it's only allowed to pick one or two frames to represent an entire video. In those conditions, the difference between a smart selection and a random one is the difference between a correct description and a completely wrong one. PEEK consistently outperforms uniform sampling and other adaptive methods in those constrained scenarios.
Alex: So the real payoff isn't just speed. It's that the model's limited attention is being spent on the right things.
Sam: That's a good way to put it. The fundamental problem with video-language models is that they can only look at so much. PEEK doesn't change that limit — it just makes sure that whatever the model does look at is as useful as possible. By training a fast student to carry the teacher's judgement, the researchers have found a practical way to close the gap between being efficient and being intelligent.
Alex: Thanks for walking us through this, Sam. And thanks to our listeners for joining us on ResearchPod.