Long-horizon omnimodal question answering answers questions by reasoning over text, images, audio, and video. Despite recent progress on OmniLLMs, low-resource long audio-video QA still suffers from costly dense encoding, weak fine-grained retrieval, limited proactive planning, and no clear end-to-end optimization. To address these issues, we propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning. It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks. Moreover, it uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries. Furthermore, we apply group relative policy optimization to jointly improve tool use and answer quality over time. Experiments on OmniVideoBench, WorldSense, and Daily-Omni show that OmniRAG-Agent consistently outperforms prior methods under low-resource settings and achieves strong results, with ablations validating each component.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a paper called OmniRAG-Agent. It describes a way for AI to answer questions from long videos and audio clips—like 20 to 30 minutes long—without using huge amounts of computer memory.
Alex: So the main problem is that normal AI models run out of memory when processing full-length videos?
Sam: Yes. These models handle text, pictures, sounds, and moving images together. But they have to process every frame and sound at once, which overloads memory. In real tasks, like figuring out why a badminton player argued with a referee in a 20-minute match, key details—like a quick comment at 30 seconds or a gesture at 53 seconds—are scattered and small. Current methods either compress the whole video and lose details or use massive compute to scan everything.
Alex: Like trying to watch a whole movie without skipping scenes?
Sam: The paper fixes this with external storage for key snippets. It works on standard setups with just 15 gigabytes of video memory—about a quarter of what others need. First, it shrinks the full video to a quick overview that fits in memory. This includes key images pulled every few seconds and speech-to-text chunks from the audio—like jotting down chapter notes from a movie.
Alex: So it's like a highlight reel instead of the full game tape?
Sam: Yes. It builds two separate collections: an image bank with those frames and time labels, and an audio bank with text pieces. The AI acts like a detective. It asks targeted questions to search these banks using similarity matches—like comparing a description to video frames to find the best fits. This feeds only the needed evidence back, without overloading memory.
Alex: How does it know which highlights to pull, especially for tricky questions needing multiple clues?
Sam: It loops through planning, retrieving, and thinking turns until the answer forms. To get better at this, they train it with reinforcement learning—like replaying a video game run and scoring the outcome. Good runs get reinforced; poor ones get dialed back. They score on two things: sticking to a strict format for queries and answers, and matching the final answer exactly. No credit for correct answers if the steps are sloppy.
Alex: Like treats for winning levels, but penalties for messing up?
Sam: Precisely. They rank rewards relative to a group of practice runs—like grading on a curve—and nudge probabilities smoothly. They cap big changes to avoid drifting too far. The paper calls this Group Relative Policy Optimization, or GRPO.
Alex: Does this training pay off?
Sam: Yes. Ablations show it gives a consistent lift across benchmarks, especially for open-source models. On OmniVideoBench, the full setup outperforms the version without GRPO in reasoning and perception tasks. It generalizes to different AI backbones or datasets.
Alex: So it's learning to retrieve reliably over time. That seems like a meaningful step for long videos on everyday hardware.
Sam: The agent queries the banks one at a time with plain descriptions, pulling the top few closest matches—like searching a library catalog. The matching system compares the query to each snippet's content and returns the best ones with exact times. This stays under 15 gigabytes by grabbing only what's needed, avoiding over 60 for the full video.
Alex: Like jumping to timestamps in a video player?
Sam: Yes. On benchmarks with 20-to-30 minute clips, it reasons about twice as effectively as baselines that compress everything. The authors caution it depends on search quality, can accumulate errors over steps, and adds some compute time.
Alex: Does the paper explain mathematically why retrieval plus loops outperform simpler methods?
Sam: Yes, with proofs for three parts. First, for scattered clues—like a quick referee shout—retrieval pulls at least one vital piece. Without it, the AI uses only a blurry summary. Accuracy boosts by at least the retrieval chance times the evidence payoff.
Alex: Like peeking at plot twists to guess the ending?
Sam: Exactly. Using both banks stacks the odds, since clues split across sight and sound. Loops shrink uncertainty each turn—like piecing a puzzle where early pieces guide better grabs. Training nudges toward high-scoring full runs using safe group comparisons.
Alex: That ties the wins to solid reasoning: retrieval odds, loop refinement, tuned habits.
Sam: The paper notes limits. It relies on speech-to-text for the audio bank—if noisy from accents or background, retrieval misses clues. Image searches use embeddings, numerical fingerprints of pictures—if they miss subtle details, matches falter. Multi-turn risks error buildup, and budgets need tweaks.
Alex: Anything on real-world use, like on consumer gear?
Sam: On OmniVideoBench, it scores about twice as high as baselines on tasks needing scattered sight-and-sound links, like spotting arguments in sports footage, using far less memory.
Alex: A practical way to handle extended audio-video on everyday computers, with limits noted. Thanks, Sam, for breaking it down. That's our look at OmniRAG-Agent. Thanks for listening to ResearchPod.