MEDit-Bench: A Dataset for Evaluating Message-Driven Narrative Video Editing

Video editing is a creative process where the same source footage can be transformed into vastly different narratives depending on the editor's goal. Traditional video summarization benchmarks treat editing as a task of identifying visually salient moments, effectively ignoring the 'message' or intent behind the edit. This paper introduces MEDit-Bench, a dataset and benchmark designed to evaluate how well models can perform message-driven video editing.

The authors curated 60 long-form videos (7–18 minutes each) and paired them with 180 distinct narrative messages. For each pair, professional editors produced multiple edits, resulting in 540 reference videos. The researchers established an evaluation protocol using temporal alignment metrics (such as Recall and F1 at various IoU thresholds) and a human perceptual study. They also developed 'MEDitor,' a model trained using Group Relative Policy Optimization (GRPO) with rewards that account for both temporal coverage and cut-level precision.

The study reveals that editorial intent is a critical factor in video construction; different messages lead to substantially different shot sequences even when the source footage is identical. While strong models like Gemini-3-pro and GPT-5 show promise in temporal alignment at lenient thresholds, they consistently fall behind human editors when stricter criteria are applied. Furthermore, the authors demonstrate that message difficulty—measured by ambiguity and contextfulness—is a reliable predictor of model performance, with more complex narratives leading to lower accuracy.

By moving beyond simple saliency-based summarization, this work provides a more realistic framework for evaluating how AI can assist in creative storytelling. The findings highlight a significant gap between current MLLMs and professional human editors, suggesting that future research must focus on deeper narrative comprehension rather than just identifying visually important segments.