Retrieval-augmented generation is moving beyond text into long, egocentric video, where systems must select query-relevant chunks across multiple modalities and temporal granularities. Yet progress in VideoRAG is limited by two gaps: existing benchmarks allow queries to be answered without the video, obscuring retrieval errors, and prior methods apply a single modality-granularity configuration per query, ignoring chunk-level variability. We address both by introducing V-RAGBench, a benchmark of $\langle$query, evidence chunk, answer$\rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk. Each chunk then enters the generator under its winning configuration selected during retrieval, yielding an interleaved evidence form where the chunk-level decision propagates across both stages. CARVE outperforms eight recent VideoRAG baselines, with the chunks supplied to the generator interleaving multiple configurations rather than sharing a single one, a behavior unattainable by query-level methods.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that rethinks how AI searches through long videos to answer questions.
Sam: So this is basically making AI act like a librarian for video archives?
Alex: That's a good way to put it. The problem is that current systems treat video like a static document — they pick one fixed way of reading it and apply that same approach to every single moment. But video is more varied than that. A cooking tutorial has moments that are best understood visually, and moments where a written summary of what the chef said is far more useful. The research argues that accuracy depends on letting each moment choose its own best representation.
Sam: So if you search for "when did I last see my keys," a generic system fails because it doesn't know whether to look at a visual frame or a text summary of what was happening?
Alex: Exactly. And the key insight is that the right way to read a video chunk isn't determined by your question — it's determined by the content of that chunk itself. A moment with lots of on-screen action is best understood as an image. A moment where someone is explaining something verbally is best understood as text.
Sam: So they built a system that dynamically switches between those perspectives depending on what it's looking at?
Alex: That's the core idea. They call it CARVE — Chunk-Aware Reranking for Video Evidence. It runs four different search methods in parallel, mixing visual and text approaches at both the individual frame level and the broader clip level, and then picks the best-performing one for each specific piece of video.
Sam: How does it actually decide which approach is right for a given moment?
Alex: It uses what's called a cross-encoder. Think of it as a scoring judge. It looks at your question and a video chunk side by side, then evaluates each possible representation — visual frame, text summary, and so on — and assigns each one a relevance score. Whichever representation scores highest for that specific chunk is the one the system uses.
Sam: So it's not one judge making a blanket ruling for the whole video. It's a fresh judgment for every single chunk.
Alex: Precisely. And that per-chunk evaluation is what separates this from earlier approaches. Most systems pick a representation strategy once and apply it everywhere. CARVE treats every chunk as its own problem to solve.
Sam: Now, to test whether this actually works, they needed a way to measure it fairly. How did they do that?
Alex: That turned out to be its own significant challenge. They built a new benchmark called V-RAGBench. The core requirement was that the AI should only be able to answer a question correctly if it genuinely retrieved the right video evidence — not because it already knew the answer from general training. Previous datasets had a flaw: a system could guess the correct answer without ever finding the relevant video clip, which made it impossible to tell whether the search process was actually working.
Sam: So the old benchmarks were essentially giving the AI an open-book exam, but the book was optional.
Alex: That's a fair description. V-RAGBench enforces what the paper calls visual grounding — the answer has to be traceable back to a specific moment in the video. That way, retrieval and answer generation can be evaluated as separate, measurable steps for the first time.
Sam: And how did CARVE perform under those stricter conditions?
Alex: It outperformed eight recent comparison systems without requiring any additional training. The consistent finding was that mixing representation types — rather than committing to just one — produces meaningfully better results across the board.
Sam: Which suggests the problem wasn't the AI's reasoning ability. It was the quality of the evidence it was being handed.
Alex: That's a useful way to frame it. The paper's broader argument is that in video question-answering, how you represent the data matters as much as how powerful your model is. CARVE is a notable step toward building systems that are honest about what they actually found — and flexible enough to find it well. Thanks for listening to ResearchPod.