MetaVideoAgent: Automated Video-Agent Evolution for Long-Form Video Understanding

Long-form video understanding requires locating sparse, relevant evidence within extended multimodal content. Existing video agents often use fixed designs that fail to adapt to the varying information density, content structures, and evidence patterns of different video distributions. This paper asks whether an automated evolution framework can adaptively redesign video agents for specific target distributions to improve accuracy and resource efficiency.

The authors introduce MetaVideoAgent, a framework that automates the evolution of video agents through three iterative stages:

1. Distribution-Aware Design: Profiles modality-specific information density from sparsely sampled frames to guide the initial agent architecture.
2. Teacher-Led Diagnosis: Uses ground-truth evidence intervals to construct a 'Gold Path,' auditing student trajectories to identify the earliest point of failure and isolating these into minimal, independently executable validation tasks.
3. Modular Evolution: Restricts code-level updates to the primary responsible module identified during diagnosis, ensuring that changes are targeted and maintain system stability.

The authors also introduce VA-EvoBench, a benchmark comprising eight distinct video distributions (e.g., e-commerce, sports, narrative) with separate evolution and held-out splits to evaluate the framework's performance.

MetaVideoAgent significantly outperforms existing fixed-design video agents and direct MLLM baselines. Across eight diverse video distributions, the framework raised the macro-average accuracy from 38.44% to 51.47%. Notably, these evolved agents achieved higher accuracy while simultaneously reducing the average number of tokens and video frames processed per question, demonstrating that distribution-specific optimization leads to both higher performance and greater computational efficiency.