Killian Steunou, Yannis Tevissen, Mounim A. El Yacoubi
5 min
Video Large Language Models (VideoLLMs) have become the standard for video understanding, enabling open-ended reasoning and instruction following. However, their reliance on processing high-resolution frames and long temporal contexts leads to significant computational and memory overhead. This survey provides a comprehensive synthesis of inference-efficiency mechanisms designed to reduce parameter counts, FLOPs, latency, and memory usage in these models.
The authors decompose the VideoLLM architecture into a four-stage pipeline:
The survey highlights that while many methods focus on a single stage, the most successful recent approaches combine techniques across the pipeline. For instance, methods that prune tokens before the LLM and then apply KV-cache management within the decoder layers often achieve the best trade-offs. The authors also note that efficiency is not just about reducing cost; saved compute is frequently reinvested to increase temporal coverage, allowing models to process longer videos without increasing the total inference budget.
A major barrier to progress is the lack of a standardized evaluation protocol. Current research relies on heterogeneous benchmarks and hardware setups, making it difficult to determine which mechanisms are truly superior. The authors advocate for a common protocol that fixes the host model, input protocol, and FLOP-accounting boundaries. Furthermore, they identify audiovisual efficiency—specifically how audio can guide visual token reduction—as a promising but under-explored area.
Video understanding has rapidly evolved toward video large language models (VideoLLMs): systems that couple video representations with pretrained large language models and condition generation on a textual prompt. Their strong performance on captioning, question answering, retrieval and temporal grounding comes at a computation and memory cost that grows with frame count and context length, limiting deployment in real-time, mobile and resource-constrained settings. This survey covers inference-efficiency mechanisms for visual and audiovisual VideoLLMs that report concrete reductions in parameter count, FLOPs per input, latency, memory, or visual and audio token count. We analyze bottlenecks across frame sampling, modality encoding, connector-level token reduction, and LLM prefilling and decoding. We organize methods by the pipeline stage at which they act, covering VideoLLMs developed since late 2022 together with earlier frame-sampling and vision-encoder mechanisms that remain components of current pipelines. We assemble literature-reported accuracy--cost comparisons under shared host models and input protocols wherever available, distinguish them from heterogeneous cross-paper evidence, and identify gaps in audiovisual efficiency and standardized evaluation. We maintain a repository at https://github.com/momentslab/awesome-efficient-videollm.
Sam: [reflective] So aggregate benchmark performance can mask domain-specific failures entirely. A method looks efficient on average, but that average is dominated by the easy cases.
Alex: [deliberate] Which is why the authors advocate for per-domain reporting alongside standardized FLOP accounting across the full encoder-connector-LLM chain. They point to the LLaVA-OneVision-7B setup as a practical starting point — fix the host model and input protocol, and you can actually isolate whether a method improves efficiency or just shifts the bottleneck somewhere less visible.
Sam: [processing] That's a meaningful methodological ask. It requires the community to treat efficiency as a first-class metric with the same rigor we'd apply to accuracy — not just a secondary result reported in an ablation table. [[RP_SECTION:audiovisual-integration-challenges|Audiovisual Integration Challenges]]
Alex: [nodding] And the stakes get higher as inputs get richer. The survey identifies audiovisual efficiency as the most significant open gap. Most of the literature is still visual-centric, but long-form video with synchronized audio introduces a joint token budget problem that current systems aren't designed to handle. Audio is a dense, temporally structured signal — it carries speaker identity, prosody, environmental context — and right now it's largely treated as an afterthought.
Sam: [leaning in] So if you compress the video stream but leave audio processing untouched, you've just relocated the bottleneck rather than eliminated it.
Alex: [measured] Exactly. And the inverse is also true — audio could actually be used to guide token reduction in the visual stream. If the audio signal indicates a static scene, you don't need high frame-rate visual sampling. That kind of cross-modal coordination isn't happening yet at the systems level.
Sam: [reflective] That's a genuinely underexplored direction. The modalities could inform each other's compression strategies rather than being optimized independently. [[RP_SECTION:future-pipeline-engineering|Future Pipeline Engineering]]
Alex: [concluding with quiet confidence] That's the broader shift the survey is pointing toward — from isolated efficiency tricks to something closer to pipeline engineering. The contribution here isn't a new method; it's a diagnostic framework. It gives you the vocabulary to ask: which stage is the constraint, what is its scaling profile, and which class of intervention addresses that specific profile? Without that structure, you're just applying optimizations and hoping the latency drops.
Sam: [grounded] And the honest read of where the field stands is that most published work is still in the "hoping" phase — good results on narrow benchmarks, without the cross-stage accounting needed to know whether the gains are real or just redistributed.
Alex: [measured] That's a fair characterization. The survey's value is in making that gap legible. Whether the community adopts the standardized evaluation protocol it proposes is a separate question — but the diagnostic map is now on the table. Thanks for listening to ResearchPod.