Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new paper called EarlyTom, which tackles a specific problem in how AI models "watch" video.
Sam: Thanks, Alex. The central puzzle is that modern video AI systems are often slow to start answering questions. These systems work in two stages: first, a "vision encoder" translates raw video pixels into a format the system can reason about, and then a language model uses that information to generate a response. The delay happens because the vision encoder is processing an enormous amount of visual data before the language model can even begin.
Alex: So the bottleneck is right at the start—the model is doing all this heavy lifting just to understand what it's looking at, before it's said a single word?
Sam: Exactly. Researchers call this delay "Time-To-First-Token"—it's simply how long you wait before the model starts typing its answer. Most previous work tried to speed things up by trimming data *after* the vision encoder had already finished its job. This paper argues that's too late. The waste has already happened.
Alex: That makes intuitive sense. If you wait until the end to throw away the filler, you've already spent the time and energy processing it.
Sam: Right. Think about a video of someone sitting at a desk, mostly still, occasionally typing. Frame after frame looks nearly identical. But the encoder dutifully processes every single one as if it were brand new. That's where the time goes.
Alex: It's like transcribing a meeting word for word, when you could have just noted the new points and skipped the repetitive parts.
Sam: That's a good way to put it. And the researchers found that the vision encoder alone accounts for more than half of the total wait time. So that's where they focused their effort.
Alex: How do they actually identify which frames are repetitive? Is it just a rough estimate?
Sam: It's more principled than that. They track how much each frame differs from the ones before it, using a technique that gives more weight to recent frames than older ones—so the system is always comparing against a kind of rolling average of recent visual history. If a new frame looks very similar to that recent history, it gets flagged as redundant and merged with its neighbors rather than processed separately. This is what the paper calls "Inner-Vision Encoder Frame Merging."
Alex: So the system is constantly asking: "Is anything actually new here?" And if the answer is no, it skips the redundant work.
Sam: Precisely. But that only solves half the problem. Even within a single frame, not everything matters equally. A busy street scene might have a lot of movement in one corner and a blank wall in another. If you compress the whole frame uniformly, you lose the important details.
Alex: So how do they handle that?
Sam: This is their second key idea. They separate the visual information into two categories: parts of the scene that are actively changing—dynamic regions—and parts that are mostly still—static regions. For the dynamic parts, they look across the whole frame to find what's most important. For the static parts, they use a more local approach, only comparing nearby regions to each other.
Alex: Why does that local approach matter for the static parts specifically?
Sam: This is where something subtle comes in. The model has a structural quirk: certain areas of a frame—often the edges or corners—consistently attract high attention scores, regardless of whether anything interesting is actually happening there. The paper calls these "sink tokens." They're a bit like a visual habit the model can't shake.
Alex: So they're like a distraction the model keeps returning to, even when there's nothing important there?
Sam: Exactly. If you look across the whole frame when selecting the important static regions, these sink tokens can crowd out the genuinely meaningful content—the model mistakes its own habitual focus for real importance. By keeping the selection local for static regions, the researchers ensure those structural attractors don't skew the results. You only compare a region to its immediate neighbors, so a noisy edge can't outcompete something meaningful happening elsewhere in the frame.
Alex: So they're not just compressing the data—they're actively correcting for a bias in the model's own attention.
Sam: That's a precise way to describe it. And the combination of both techniques—merging redundant frames early, then carefully selecting what matters within each remaining frame—produces a meaningful improvement in practice. The start-up delay drops by more than half, and the overall computational work is reduced substantially, all without retraining the model.
Alex: That last point seems important. They're not building something from scratch—they're making an existing system smarter about how it uses what it already knows.
Sam: Exactly. That's what makes this approach practical. The underlying model doesn't change. You're just giving it a smarter set of eyes—one that knows when to look carefully and when to move on. The broader lesson the paper draws is that *where* you compress matters as much as *how much* you compress. Doing it early, inside the encoder, before the bottleneck forms, turns out to be far more effective than cleaning up afterward.
Alex: It's a good reminder that in complex systems, the most effective fix is often the one closest to the source of the problem. Thanks for listening to ResearchPod.