GraphVid: Interactive Graph-Controllable Video Generation

Controllable video generation is often limited by the difficulty of specifying complex multi-object interactions. Existing methods typically rely on trajectory-based inputs or dense motion cues, which are cumbersome to author and scale poorly. GraphVid addresses this by asking: can we use structured, human-interpretable interaction graphs to provide flexible, precise control over multi-object video generation?

GraphVid introduces a graph-conditioned framework that translates user-defined interaction graphs into conditioning tokens for a frozen video diffusion transformer. The process begins by constructing a scene graph from an input image, where nodes represent entities and directed edges represent specific relational interactions (e.g., 'pull', 'holding', 'leaning').

To process these graphs, the authors propose an Edge-Aware Graph Reasoning module based on GINEConv layers. This module explicitly incorporates edge attributes into the message-passing process, allowing the model to understand how specific relational roles influence object motion. These interaction-aware embeddings are then mapped into the latent space of a pretrained video diffusion model using lightweight LoRA adapters. This design allows the model to learn complex, interaction-driven dynamics while preserving the high-quality generative priors of the frozen backbone.

GraphVid demonstrates that structured semantic interfaces are a powerful paradigm for video control. In empirical evaluations, the model significantly outperforms state-of-the-art trajectory-based baselines like Motion-I2V. Specifically, GraphVid reduces FID by up to 39.9% and FVD by 37.6%, while simultaneously improving PSNR and SSIM metrics. These gains are achieved with substantially fewer trainable parameters (0.6B) and less training data, highlighting the efficiency of graph-based conditioning over dense motion-field supervision.