NEST: Narrative Event Structures in Time for Long Video Understanding

Research Context and Motivation

Recent advancements in vision-language models (VLMs) have enabled the processing of longer video sequences, but this capacity has not translated into a deeper understanding of narrative structure. Current benchmarks for long-form video often prioritize simple retrieval tasks or atomic action recognition in short clips, failing to capture how complex events interact, evolve, or relate to one another over the course of a full-length film. Understanding a narrative requires reasoning over long temporal gaps, such as connecting an early event to a later consequence despite intervening scenes or flashbacks.

The NEST Dataset

To address this gap, the authors introduce NEST (Narrative Event Structures in Time), a dataset comprising 1,005 full-length movies with an average duration of 98 minutes. Each movie is annotated with approximately 102 multimodal narrative events grounded in visual content, dialogue, and audio. The dataset captures complex narrative structures, including temporal ordering, hierarchical composition, and long-range dependencies. By leveraging audio descriptions—professional narrations designed for visually impaired audiences—the authors provide a high-quality, human-authored foundation for grounding events in both visual and textual contexts.

Benchmark Tasks and Performance

The authors define four primary tasks to evaluate narrative comprehension: Event Trigger Detection (ETD), Event Localization (EL), Event Argument Extraction (EAE), and Event Relation Extraction (ERE). The benchmark results demonstrate that these tasks are highly challenging for current state-of-the-art models. Specifically, performance for ETD is below 8%, EL is under 6%, and EAE is below 11%. While Event Relation Extraction (ERE) is more tractable once events are identified—reaching 44.42% F1 after fine-tuning—the overall difficulty of the benchmark highlights the significant gap between current model capabilities and the requirements for true narrative-level reasoning in long-form video.

NEST shifts the focus of video understanding from simple frame-level classification to high-level narrative reasoning. By providing a structured, long-form benchmark, it enables researchers to develop models that can track characters, causal chains, and thematic arcs across hours of content, which is essential for applications in education, automated video analysis, and advanced storytelling systems.