We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a new study tackling a real challenge in AI: how do we actually measure whether a video-processing model is any good?
Sam: We're discussing the Massive Video Embedding Benchmark, or MVEB. Right now, video AI is a fragmented landscape. Most models are trained and tested on one very specific task — say, recognising actions in sports clips — and they get very good at that one thing. The problem is, a high score on a narrow test doesn't tell you whether the model genuinely understands video in a broader sense.
Alex: So it's a bit like a student who's brilliant at multiple-choice questions but falls apart the moment you ask them to write an essay?
Sam: That's a fair comparison. And this study is trying to fix that by introducing a benchmark — think of it as a comprehensive exam — that covers 23 different tasks. Everything from classifying what's happening in a clip, to answering questions about video content. The goal is to get a much more honest picture of what a model can actually do.
Alex: And I understand they started with a much larger pool — 184 tasks. Why go to the trouble of filtering down to just 23?
Sam: Because running a model through 184 separate tests is genuinely expensive — in terms of computing power and time. So the researchers used a process they call redundancy pruning. The idea is to identify tasks that are essentially measuring the same underlying ability, and remove the duplicates. Think of it like a standardised exam: you don't need 100 algebra questions if 10 well-chosen ones reliably predict how a student would have done on all 100.
Alex: So they're not losing information — they're compressing it efficiently?
Sam: Exactly. And they verified this carefully. The trimmed 23-task benchmark tracks the full 184-task results with very high fidelity, so researchers can trust they're getting an accurate picture without the massive compute cost.
Alex: I want to ask about audio, because that's something I wouldn't have expected to be complicated. Does adding sound to a video always help a model understand it better?
Sam: This is one of the more interesting findings. The short answer is: it depends entirely on how the training data was labelled. If the labels — the descriptions or categories attached to each video — were created using both the visuals and the audio, then yes, sound helps the model. But if the labels were based only on what people could see, then the audio becomes noise. The model starts paying attention to something that the labels never rewarded it for noticing, and its performance drops noticeably.
Alex: So the audio isn't inherently useful or harmful — it's about whether the model was ever taught to connect sound to meaning?
Sam: Precisely. And this has a practical consequence. Developers building video AI have to make a real choice: invest in expensive audio-processing capabilities, or stick to a visual-only approach? The benchmark provides separate leaderboards for these different configurations, so teams can make that decision based on evidence rather than assumption.
Alex: Right. And once you have all this data from 23 tasks, how do you actually turn it into a single ranking? You can't just average percentages across very different types of tests.
Sam: That's where a method called the Borda count comes in. Think of it like a ranked-choice vote. Instead of looking at raw scores, you look at where each model placed on each task — first, second, third, and so on — and award points accordingly. A model that consistently finishes near the top across many different tasks accumulates more points than one that dominates a single test but struggles everywhere else.
Alex: So it rewards breadth of understanding, not just depth in one area.
Sam: Exactly. It's designed to prevent a specialist model from crowding out genuinely capable general-purpose ones on the leaderboard.
Alex: And there's also the question of zero-shot evaluation — testing models on tasks they've never specifically trained for. How does that change things?
Sam: Zero-shot testing is essentially the ultimate pop quiz. You take a model and give it a task it has never seen during training, and you measure how well it copes. It reveals whether the model has built a genuine, flexible understanding of the world, or whether it's been memorising patterns specific to its training data.
Alex: Though I imagine there's a risk that a model has technically "seen" some of the test material without anyone realising?
Sam: That's a real concern, and the study takes it seriously. They track the overlap between training data and test data — what's called train-test contamination — to flag cases where a model might be scoring well for the wrong reasons. Keeping that honest is essential for the benchmark to mean anything.
Alex: Are there areas where this benchmark still falls short?
Sam: The authors are candid about this. Right now, MVEB doesn't have deep coverage of specialised domains like medical imaging. It's a general-purpose map of the landscape, not a high-resolution scan of every niche. And it currently focuses on shorter clips rather than long-form video — testing whether a model can maintain understanding over hours of footage is a meaningful next step the paper points toward.
Alex: So what we have is a solid, practical foundation. A standardised way to compare models honestly, across a wide range of tasks, at a cost that's actually feasible for most research teams.
Sam: That's the core contribution. And by building it into an existing, community-maintained ecosystem, the authors have made it something that can grow and adapt as the field develops — rather than becoming outdated the moment a new generation of models arrives. What video AI has needed is a shared measuring stick. This is a meaningful step toward providing one.
Alex: Thanks for listening to ResearchPod.