Yuheng Huang, Jianlang Chen, Jiayang Song, Hua Qi, Aza Kai, Vincent Markert, Edison Marrese-Taylor, Jianjun Zhao, Lei Ma
5 min
Recent advances in Multimodal Large Language Models (MLLMs) have significantly improved short-form video understanding. However, many real-world applications involve context-rich long-form videos, such as full-length films, television programs, and documentaries. Understanding these media requires models to jointly perform two demanding tasks: maintaining narrative coherence over hours of footage and interpreting culturally nuanced, implicit communication. Existing benchmarks rarely evaluate these capabilities simultaneously, particularly in high-context, non-English settings.
To address this gap, the authors introduce NARU, a large-scale benchmark designed to evaluate narrative intelligence and cultural understanding in Japanese extreme long-form video. Japanese media provides an ideal testing ground due to its high-context communication style, where meaning is frequently conveyed implicitly through ambient atmosphere, conversational backchannels, and shared cultural expectations.
The NARU benchmark is structured around a comprehensive taxonomy divided into two primary dimensions. Narrative Intelligence (N) is evaluated across four subcategories: Character/Entity Evolution (N.1), Sequential/Topical Flow (N.2), Plot/Conflict Progression (N.3), and Idea/Thematic Development (N.4). Cultural Understanding (C) focuses on interactions where meaning depends on shared social assumptions, covering Aizuchi or conversational backchannels (C.1), Kuuki wo Yomu or shared situational understanding (C.2), Subtext Interpretation (C.3), Cultural Context Recognition (C.4), and Sentiment Analysis (C.5).
To construct the benchmark, the authors curated 155 Japanese long-form videos totaling 146.8 hours from an initial pool of over 10,000 candidates, ensuring high visual integrity and meaningful temporal progression. The dataset comprises 1,481 multiple-choice questions validated through a rigorous multi-stage expert verification process involving 68 native Japanese annotators.
Because manual annotation of extremely long videos is prohibitively expensive and unscalable, the authors propose a hierarchical memory-based annotation pipeline. This pipeline decomposes videos into temporal chunks, maintains cross-segment narrative continuity through structured event and cultural annotations, and generates question-answer pairs via task-oriented synthesis and iterative shortcut removal. A multi-agent refinement loop eliminates textual shortcuts to ensure questions require genuine multimodal video understanding.
Evaluating eight prominent MLLM configurations on NARU reveals substantial performance gaps. Current models struggle significantly with both long-range narrative integration and culturally grounded reasoning, highlighting that existing architectures are not yet equipped to reliably interpret high-context, long-form video content.
Long-form video understanding encompasses tasks that go beyond retrieving isolated events, including tracking an evolving narrative and interpreting social meaning that may remain implicit. However, existing benchmarks rarely evaluate these capabilities jointly, particularly in high-context, non-English media. To address this gap, we introduce NARU, a benchmark designed to evaluate Narrative evolution and Reasoning on cultural Understanding in Japanese long-form video. NARU consists of 1,481 questions grounded in 155 videos totaling 146.8 hours, spanning four narrative and five cultural dimensions. To construct the benchmark at this scale, we propose a hierarchical memory-based annotation pipeline that transforms raw video into structured event, narrative, and cultural annotations, then generates questions via task-oriented synthesis and iterative shortcut removal. The construction process includes two native-speaker verification stages involving 68 annotators. Evaluations across eight model configurations reveal substantial limitations in both long-range narrative integration and culturally grounded reasoning. By exposing these persistent gaps, NARU offers a systematic testing ground for developing MLLMs capable of reliably interpreting long-form, high-context video.
Alex: Right. And once that structured map exists, a model uses it to draft multiple-choice questions with plausible wrong answers. Crucially, all the answer options are written to look similar in length and style, so a model can't guess by spotting the odd one out.
Sam: That's a reasonable safeguard. But couldn't a model still answer correctly just by recognizing patterns in the language, without actually understanding the video?
Alex: That's exactly the problem they anticipated. So they built in what they call an iterative debiasing loop. A test agent tries to answer each question without seeing the video at all — just the question and the options. If it gets the answer right, that's a red flag. It means the question is answerable through language shortcuts alone.
Sam: And then what happens?
Alex: A second agent diagnoses why the shortcut worked, and a third rewrites the question until the blind agent can no longer guess correctly. The loop repeats until the question genuinely requires watching the video. It's a self-checking process built into the design.
Sam: So the benchmark is actively hardened against the kind of shortcut-taking AI systems are known for.
Alex: That's the intent. Now, when they actually ran models through NARU, a few things stood out. They tested both a standard multiple-choice format and an open-ended format where models had to write free-form answers rather than pick from options.
Sam: Why does that distinction matter?
Alex: Because answer choices act as a kind of scaffold. When options are there, a model can sometimes reconstruct the correct sequence of events just by comparing the choices against each other. Take that away, and the model has to rebuild the timeline entirely from its own memory. One category — tracking the sequential flow of events — was relatively manageable in multiple-choice. In the open-ended format, it became the weakest area for most models tested.
Sam: So the options were doing more of the cognitive work than it appeared.
Alex: Precisely. The other finding worth noting involves what happens when you give models more visual information. If you increase the number of video frames a model can process, its ability to follow narrative events improves — because it's less likely to miss a key moment scattered across hours of footage.
Sam: But cultural understanding doesn't get the same boost?
Alex: No, and that's the telling part. Cultural interpretation doesn't depend on seeing more frames. It depends on having the underlying social knowledge to read what those frames mean. More visual data doesn't fix a gap in that kind of reasoning. The two problems require different solutions.
Sam: You can't just scale your way out of a knowledge problem.
Alex: That's essentially the central takeaway of the paper. NARU is designed to separate two things that often get conflated: the ability to track what happens in a video, and the ability to understand what it means within a cultural context. Current models, even the strongest commercial ones, show a clear gap on the second.
Sam: It makes you wonder how much of what we call "video understanding" is actually just pattern recognition on short clips.
Alex: That's a fair question, and one this benchmark is specifically designed to push on. Thanks for listening to ResearchPod.