Video-language models can process only a limited number of frames, making frame selection a key bottleneck for efficient video captioning. Most captioning pipelines still rely on uniform sampling, which is computationally cheap but agnostic to visual content. Adaptive frame sampling has recently emerged as a promising approach for selecting the most informative frames from a video; however, existing methods remain computationally expensive. We introduce PEEK, an efficient dynamic frame sampling method that distills caption-conditioned frame relevance rankings from a stronger teacher model into a lightweight temporal model that operates only on visual content. We find that, overall, on ActivityNet Captions and MSR-VTT, our method outperforms state-of-the-art methods across all evaluated downstream vision language models, especially when only one or two frames are selected for captioning, obtaining the best CIDEr for most frame budgets. On ActivityNet Captions, PEEK is particularly strong, winning 14 out of 16 configurations. Zero-shot evaluation on MSR-VTT shows that our model transfers best at low frame budgets, while results at four and eight frames are more mixed as temporal coverage and visual diversity become increasingly competitive. Compared with recent adaptive baselines, PEEK is both more accurate in the low-budget regime and more efficient: it adds only $5.2\%$ to the captioning time, compared with $65.4\%$ for CSTA and $211.9\%$ for MaxInfo. We release our code and pre-trained checkpoint at https://github.com/momentslab/peek.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a challenge in AI: how to make video-processing models smarter about what they watch, without making them too slow to use.
Sam: That's the central puzzle. These systems — called video-language models — are designed to watch a video clip and describe what's happening in it. But they can't actually process every single frame. A one-minute video at standard speed has around 1,800 individual images. So the system has to choose which ones to look at.
Alex: And the wrong choice means it might miss the most important moment entirely.
Sam: Exactly. Most systems today use what's called uniform sampling. Think of it like setting a timer to take a photograph every five seconds while watching a football match. It's simple and cheap to run, but it's completely blind to what's actually happening on screen. You might end up with a photo of an empty corner of the pitch and miss the moment the goal goes in.
Alex: So it's efficient, but not smart. What's the alternative?
Sam: The other extreme is to have a powerful AI model examine every single frame and score it against a text description — like asking "does this frame show a goal being scored?" That's much more accurate, but it's also enormously slow. The researchers behind this paper wanted something in between: a system that's fast, but still knows how to pick the frames that actually matter.
Alex: So they're looking for a shortcut — a way to be selective without doing all that heavy work every time.
Sam: Right. And the way they built that shortcut is through a technique called knowledge distillation. Here's the idea: imagine a very experienced teacher who has read the full script of a film before watching it. Because they already know what happens, they can immediately spot which scenes are the most important. Now imagine that teacher watching the film alongside a student, pointing out the key moments. Over time, the student learns to recognise those moments on their own — without needing the script anymore.
Alex: And that's essentially what PEEK does?
Sam: That's the core of it. PEEK is the name of the system the researchers built. A powerful "teacher" AI, which has access to text captions describing the video, scores each frame based on how much useful information it contains. A simpler, faster "student" model watches the same video and learns to replicate those scores. Once the training is done, the teacher is removed entirely. The student can now look at a brand new video and immediately identify the most informative frames — without any text input at all.
Alex: So the student has internalised the teacher's judgement. It doesn't need to be told what to look for anymore.
Sam: Precisely. And that's why it's so much faster in practice. Other adaptive methods require running a heavy AI model over the entire video every single time you want to process something. PEEK's student model adds only around five percent to the total processing time. It's lightweight, but it's still making informed choices rather than blind ones.
Alex: That said — if the system is being that selective, isn't there a risk it just keeps picking the same kind of frame over and over? Like, always grabbing the most visually dramatic moment at the start of a video?
Sam: That's a genuine concern, and the researchers addressed it directly. They use a technique called stratified argmax. The name sounds technical, but the idea is straightforward. Imagine dividing a video into equal chapters — say, ten sections of equal length. The model is required to pick exactly one frame from each chapter, choosing whichever is the most informative within that section. It can't just grab ten frames from the opening minute and ignore the rest.
Alex: So it's a built-in rule that forces the system to spread its attention across the whole video.
Sam: Exactly. It balances picking the best available content in any given moment with making sure the whole timeline is represented. You get quality and coverage, rather than having to choose between them.
Alex: And does it actually perform better than just sampling uniformly?
Sam: The results suggest it does, particularly when the system is working with a very tight frame budget — meaning it's only allowed to pick one or two frames to represent an entire video. In those conditions, the difference between a smart selection and a random one is the difference between a correct description and a completely wrong one. PEEK consistently outperforms uniform sampling and other adaptive methods in those constrained scenarios.
Alex: So the real payoff isn't just speed. It's that the model's limited attention is being spent on the right things.
Sam: That's a good way to put it. The fundamental problem with video-language models is that they can only look at so much. PEEK doesn't change that limit — it just makes sure that whatever the model does look at is as useful as possible. By training a fast student to carry the teacher's judgement, the researchers have found a practical way to close the gap between being efficient and being intelligent.
Alex: Thanks for walking us through this, Sam. And thanks to our listeners for joining us on ResearchPod.