Yifan Zhu, Xinyu Mu, Tao Feng, Zhonghong Ou, Yuning Gong, Haoran Luo
5 min
Abstract
Long-horizon omnimodal question answering answers questions by reasoning over text, images, audio, and video. Despite recent progress on OmniLLMs, low-resource long audio-video QA still suffers from costly dense encoding, weak fine-grained retrieval, limited proactive planning, and no clear end-to-end optimization. To address these issues, we propose OmniRAG-Agent, an agentic omnimodal QA method for budgeted long audio-video reasoning. It builds an image-audio retrieval-augmented generation module that lets an OmniLLM fetch short, relevant frames and audio snippets from external banks. Moreover, it uses an agent loop that plans, calls tools across turns, and merges retrieved evidence to answer complex queries. Furthermore, we apply group relative policy optimization to jointly improve tool use and answer quality over time. Experiments on OmniVideoBench, WorldSense, and Daily-Omni show that OmniRAG-Agent consistently outperforms prior methods under low-resource settings and achieves strong results, with ablations validating each component.
Alex: Does this training pay off?
Sam: Yes. Ablations show it gives a consistent lift across benchmarks, especially for open-source models. On OmniVideoBench, the full setup outperforms the version without GRPO in reasoning and perception tasks. It generalizes to different AI backbones or datasets.
Alex: So it's learning to retrieve reliably over time. That seems like a meaningful step for long videos on everyday hardware.
Sam: The agent queries the banks one at a time with plain descriptions, pulling the top few closest matches—like searching a library catalog. The matching system compares the query to each snippet's content and returns the best ones with exact times. This stays under 15 gigabytes by grabbing only what's needed, avoiding over 60 for the full video.
Alex: Like jumping to timestamps in a video player?
Sam: Yes. On benchmarks with 20-to-30 minute clips, it reasons about twice as effectively as baselines that compress everything. The authors caution it depends on search quality, can accumulate errors over steps, and adds some compute time.
Alex: Does the paper explain mathematically why retrieval plus loops outperform simpler methods?
Sam: Yes, with proofs for three parts. First, for scattered clues—like a quick referee shout—retrieval pulls at least one vital piece. Without it, the AI uses only a blurry summary. Accuracy boosts by at least the retrieval chance times the evidence payoff.
Alex: Like peeking at plot twists to guess the ending?
Sam: Exactly. Using both banks stacks the odds, since clues split across sight and sound. Loops shrink uncertainty each turn—like piecing a puzzle where early pieces guide better grabs. Training nudges toward high-scoring full runs using safe group comparisons.
Alex: That ties the wins to solid reasoning: retrieval odds, loop refinement, tuned habits.
Sam: The paper notes limits. It relies on speech-to-text for the audio bank—if noisy from accents or background, retrieval misses clues. Image searches use embeddings, numerical fingerprints of pictures—if they miss subtle details, matches falter. Multi-turn risks error buildup, and budgets need tweaks.
Alex: Anything on real-world use, like on consumer gear?
Sam: On OmniVideoBench, it scores about twice as high as baselines on tasks needing scattered sight-and-sound links, like spotting arguments in sports footage, using far less memory.
Alex: A practical way to handle extended audio-video on everyday computers, with limits noted. Thanks, Sam, for breaking it down. That's our look at OmniRAG-Agent. Thanks for listening to ResearchPod.