Adnan El Assadi, Roman Solomatin, Isaac Chung, Chenghao Xiao, Deep Shah, Manan Dey, Shriya Sudhakar, Zacharie Bugaud, Wissam Siblini, Ayush Sunil Munot, Yashwanth Devavarapu, Rakshitha Ireddi, Michelle Yang, Márton Kardos, Niklas Muennighoff, Kenneth Enevoldsen
5 min
Abstract
We introduce the Massive Video Embedding Benchmark (MVEB), a 23-task benchmark for video embeddings spanning classification, zero-shot classification, clustering, pair classification, retrieval, and video-centric question answering. We evaluate 33 models and find that no single model dominates: MLLM-based embeddings lead on classification, clustering, pair classification, and QA; multimodal binding leads on retrieval and zero-shot classification; generative MLLMs without contrastive adaptation collapse on cross-modal tasks. Paired video-only vs. audio+video evaluations show that audio's contribution depends on dataset annotation provenance: audio helps when labels were produced from both modalities and hurts when they were produced from visuals alone, a six-point gap consistent across model families. MVEB is derived from MVEB+, a 184-task pool, and is designed to maintain task diversity while reducing evaluation cost. It integrates into the MTEB ecosystem for unified evaluation across text, image, audio, and video. We release MVEB and all 184 tasks along with code and a leaderboard at https://github.com/embeddings-benchmark/mteb.
Sam: Precisely. And this has a practical consequence. Developers building video AI have to make a real choice: invest in expensive audio-processing capabilities, or stick to a visual-only approach? The benchmark provides separate leaderboards for these different configurations, so teams can make that decision based on evidence rather than assumption.
Alex: Right. And once you have all this data from 23 tasks, how do you actually turn it into a single ranking? You can't just average percentages across very different types of tests.
Sam: That's where a method called the Borda count comes in. Think of it like a ranked-choice vote. Instead of looking at raw scores, you look at where each model placed on each task — first, second, third, and so on — and award points accordingly. A model that consistently finishes near the top across many different tasks accumulates more points than one that dominates a single test but struggles everywhere else.
Alex: So it rewards breadth of understanding, not just depth in one area.
Sam: Exactly. It's designed to prevent a specialist model from crowding out genuinely capable general-purpose ones on the leaderboard.
Alex: And there's also the question of zero-shot evaluation — testing models on tasks they've never specifically trained for. How does that change things?
Sam: Zero-shot testing is essentially the ultimate pop quiz. You take a model and give it a task it has never seen during training, and you measure how well it copes. It reveals whether the model has built a genuine, flexible understanding of the world, or whether it's been memorising patterns specific to its training data.
Alex: Though I imagine there's a risk that a model has technically "seen" some of the test material without anyone realising?
Sam: That's a real concern, and the study takes it seriously. They track the overlap between training data and test data — what's called train-test contamination — to flag cases where a model might be scoring well for the wrong reasons. Keeping that honest is essential for the benchmark to mean anything.
Alex: Are there areas where this benchmark still falls short?
Sam: The authors are candid about this. Right now, MVEB doesn't have deep coverage of specialised domains like medical imaging. It's a general-purpose map of the landscape, not a high-resolution scan of every niche. And it currently focuses on shorter clips rather than long-form video — testing whether a model can maintain understanding over hours of footage is a meaningful next step the paper points toward.
Alex: So what we have is a solid, practical foundation. A standardised way to compare models honestly, across a wide range of tasks, at a cost that's actually feasible for most research teams.
Sam: That's the core contribution. And by building it into an existing, community-maintained ecosystem, the authors have made it something that can grow and adapt as the field develops — rather than becoming outdated the moment a new generation of models arrives. What video AI has needed is a shared measuring stick. This is a meaningful step toward providing one.
Alex: Thanks for listening to ResearchPod.