Rethinking RAG in Long Videos: What to Retrieve and How to Use It? | Yuho Lee et al. | ResearchPod

Transcript: Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

Alex: Welcome to another episode of ResearchPod. Today we're looking at a study that rethinks how AI searches through long videos to answer questions.

Sam: So this is basically making AI act like a librarian for video archives?

Alex: That's a good way to put it. The problem is that current systems treat video like a static document — they pick one fixed way of reading it and apply that same approach to every single moment. But video is more varied than that. A cooking tutorial has moments that are best understood visually, and moments where a written summary of what the chef said is far more useful. The research argues that accuracy depends on letting each moment choose its own best representation.

Sam: So if you search for "when did I last see my keys," a generic system fails because it doesn't know whether to look at a visual frame or a text summary of what was happening?

Alex: Exactly. And the key insight is that the right way to read a video chunk isn't determined by your question — it's determined by the content of that chunk itself. A moment with lots of on-screen action is best understood as an image. A moment where someone is explaining something verbally is best understood as text.

Sam: So they built a system that dynamically switches between those perspectives depending on what it's looking at?

Alex: That's the core idea. They call it CARVE — Chunk-Aware Reranking for Video Evidence. It runs four different search methods in parallel, mixing visual and text approaches at both the individual frame level and the broader clip level, and then picks the best-performing one for each specific piece of video.

Sam: How does it actually decide which approach is right for a given moment?

Alex: It uses what's called a cross-encoder. Think of it as a scoring judge. It looks at your question and a video chunk side by side, then evaluates each possible representation — visual frame, text summary, and so on — and assigns each one a relevance score. Whichever representation scores highest for that specific chunk is the one the system uses.

Sam: So it's not one judge making a blanket ruling for the whole video. It's a fresh judgment for every single chunk.

Alex: Precisely. And that per-chunk evaluation is what separates this from earlier approaches. Most systems pick a representation strategy once and apply it everywhere. CARVE treats every chunk as its own problem to solve.

Sam: Now, to test whether this actually works, they needed a way to measure it fairly. How did they do that?

Alex: That turned out to be its own significant challenge. They built a new benchmark called V-RAGBench. The core requirement was that the AI should only be able to answer a question correctly if it genuinely retrieved the right video evidence — not because it already knew the answer from general training. Previous datasets had a flaw: a system could guess the correct answer without ever finding the relevant video clip, which made it impossible to tell whether the search process was actually working.

Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

Yuho Lee, Jisu Shin, Nicole Hee-Yeon Kim, Jihwan Bang, Juntae Lee, Kyuwoong Hwang, Fatih Porikli, Hwanjun Song

Jun 11, 2026·3:36·arxiv·Original Paper

Retrieval-augmented generation (RAG) for long videosVideoRAGV-RAGBenchCARVEMultimodal and multi-granularity retrievalChunk-Aware Reranking

Listen

4 min

Briefing

Abstract

Retrieval-augmented generation is moving beyond text into long, egocentric video, where systems must select query-relevant chunks across multiple modalities and temporal granularities. Yet progress in VideoRAG is limited by two gaps: existing benchmarks allow queries to be answered without the video, obscuring retrieval errors, and prior methods apply a single modality-granularity configuration per query, ignoring chunk-level variability. We address both by introducing V-RAGBench, a benchmark of $\langle$query, evidence chunk, answer$\rangle$ triplets that enables faithful, decoupled evaluation of retrieval and generation, and CARVE, a simple method that runs parallel retrievers across configurations and employs chunk-adaptive reranking to identify the winning configuration for each chunk. Each chunk then enters the generator under its winning configuration selected during retrieval, yielding an interleaved evidence form where the chunk-level decision propagates across both stages. CARVE outperforms eight recent VideoRAG baselines, with the chunks supplied to the generator interleaving multiple configurations rather than sharing a single one, a behavior unattainable by query-level methods.

Rethinking RAG in Long Videos: What to Retrieve and How to Use It?: Research Paper Podcast

Rethinking RAG in Long Videos: What to Retrieve and How to Use It?

Listen