Hesong Wang, Xin Jin, Lu Lu, Chenhaowen Li, Jian Chen, Qiang Liu, Huan Wang
5 min
Abstract
Video large language models (Video-LLMs) have demonstrated strong capabilities in video understanding tasks. However, their practical deployment is still hindered by the inefficiency introduced by processing massive amounts of visual tokens. Although recent approaches achieve extremely low token retention ratios while maintaining accuracy comparable to full-token baselines, most of them perform compression only at the late stage of prefilling, leaving the efficiency of the vision encoder unoptimized. In this paper, we first show that vision encoding contributes a large portion to the time-to-first-token (TTFT). Therefore, instead of compressing visual tokens only after the vision encoder, performing compression inside the encoder still leaves substantial room for exploration. Based on this insight, we propose EarlyTom, a training-free token compression framework that performs early-stage visual token compression inside the vision encoder, enabling significantly better TTFT reduction and higher throughput. In addition, we introduce a decoupled spatial token selection strategy that improves the overall compression effectiveness. EarlyTom reduces TTFT by up to 2.65x and FLOPs by up to 61% on a single NVIDIA A100 GPU for the LLaVA-OneVision-7B model, while maintaining accuracy comparable to the full-token baseline. These improvements substantially enhance the practicality of deploying Video-LLMs in real-world production scenarios.
Alex: So how do they handle that?
Sam: This is their second key idea. They separate the visual information into two categories: parts of the scene that are actively changing—dynamic regions—and parts that are mostly still—static regions. For the dynamic parts, they look across the whole frame to find what's most important. For the static parts, they use a more local approach, only comparing nearby regions to each other.
Alex: Why does that local approach matter for the static parts specifically?
Sam: This is where something subtle comes in. The model has a structural quirk: certain areas of a frame—often the edges or corners—consistently attract high attention scores, regardless of whether anything interesting is actually happening there. The paper calls these "sink tokens." They're a bit like a visual habit the model can't shake.
Alex: So they're like a distraction the model keeps returning to, even when there's nothing important there?
Sam: Exactly. If you look across the whole frame when selecting the important static regions, these sink tokens can crowd out the genuinely meaningful content—the model mistakes its own habitual focus for real importance. By keeping the selection local for static regions, the researchers ensure those structural attractors don't skew the results. You only compare a region to its immediate neighbors, so a noisy edge can't outcompete something meaningful happening elsewhere in the frame.
Alex: So they're not just compressing the data—they're actively correcting for a bias in the model's own attention.
Sam: That's a precise way to describe it. And the combination of both techniques—merging redundant frames early, then carefully selecting what matters within each remaining frame—produces a meaningful improvement in practice. The start-up delay drops by more than half, and the overall computational work is reduced substantially, all without retraining the model.
Alex: That last point seems important. They're not building something from scratch—they're making an existing system smarter about how it uses what it already knows.
Sam: Exactly. That's what makes this approach practical. The underlying model doesn't change. You're just giving it a smarter set of eyes—one that knows when to look carefully and when to move on. The broader lesson the paper draws is that *where* you compress matters as much as *how much* you compress. Doing it early, inside the encoder, before the bottleneck forms, turns out to be far more effective than cleaning up afterward.
Alex: It's a good reminder that in complex systems, the most effective fix is often the one closest to the source of the problem. Thanks for listening to ResearchPod.