VideoAgent: All-in-One Framework for Video Understanding and Editing

Video editing is a labor-intensive process requiring technical expertise and multi-step workflows. While Large Language Models (LLMs) have improved, existing automated systems struggle with long-form narrative coherence and the diverse, multi-step operations required for professional-grade editing. VideoAgent addresses these gaps by providing an all-in-one agentic framework that integrates specialized tools for both video comprehension and complex editing.

VideoAgent functions through two primary innovations. First, it implements an automated shot creation system that uses planning agents to generate structured storyboards. These storyboards guide a cross-modal retrieval module that selects and trims relevant video clips from a library, ensuring the final output aligns with the user's narrative intent.

Second, the framework employs a multi-agent orchestration system. Instead of relying on a single-pass workflow generation, VideoAgent uses a technique called textual-gradient graph optimization. This method treats the construction of an editing pipeline as an iterative optimization problem. By evaluating the graph against structural and semantic quality metrics, the system uses natural language feedback to refine the workflow, removing invalid dependencies and ensuring all required editing intents are covered.

Experimental results on the newly introduced VideoEdit benchmark demonstrate that VideoAgent significantly outperforms existing multimodal LLMs and agentic systems. It achieves orchestration success rates between 87% and 95% while simultaneously reducing API costs by approximately 60%. Human evaluations across six distinct video categories indicate that the content produced by VideoAgent is of professional quality, with ratings only 4% lower than those of human-created videos. This framework effectively lowers the barrier to entry for high-quality video production by automating the planning, retrieval, and execution phases of the editing process.