Keyu Tu, Zhuowei Chen, Mengqi Huang, Yuxin Wang, Jiahao Zhu, Zhendong Mao, Yongdong Zhang
6 min
Recent advances in video generation have led to impressive gains in visual fidelity and temporal coherence. However, existing models are rarely evaluated on their ability to perform outcome-oriented tasks that require both achieving an instructed goal and preserving high-level semantic relationships with a reference image. This paper introduces Semantic Task Completion Video Generation, a task formulation where success is judged solely on the final outcome rather than the intermediate steps. To systematically study this problem, the authors present SemComp-Data, a large-scale evaluation dataset, and SemComp-Bench, a vision-language model-based evaluation protocol.
To support systematic evaluation without requiring manual task design, the authors construct SemComp-Data from full-context real-world videos. The curation pipeline consists of four main stages:
SemComp-Bench evaluates generated videos along two core dimensions using structured binary questions answered by a vision-language model. Outcome Achievement (OA) assesses outcome realization, semantic grounding, grounded entity consistency, and global visual continuity, requiring a sample to pass all four criteria for a successful OA score. Generation Reliability (GR) independently assesses physical plausibility, visual clarity, rendering artifacts, spatiotemporal coherence, and text or interface integrity. Experiments across representative open-source and proprietary video generation models reveal that while models can often generate visually appealing clips, maintaining precise reference-grounded task completion remains a significant challenge.
As video generation models transition from general-purpose aesthetic synthesis to practical, goal-oriented tools, rigorous evaluation frameworks become essential. By shifting the focus from mere visual realism to task-specific semantic grounding and outcome achievement, SemComp-Bench provides a critical diagnostic tool for measuring how well generative models understand and execute complex, instruction-guided visual transformations.
We introduce Semantic Task Completion Video Generation, an outcome-oriented video generation task. Under this formulation, success requires both achievement of the intended outcome and semantic grounding. Semantic grounding characterizes the correspondence between the reference image and the generated outcome in terms of high-level semantics relevant to the task. Evaluation focuses on the generated outcome and requires neither the presentation of a complete sequence of intermediate task steps nor conventional appearance consistency with the reference image. To support systematic evaluation, we construct SemComp-Data, an evaluation dataset covering six domains. Each instance comprises a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip. A scalable four-stage curation pipeline converts raw videos into standardized SemComp-Data instances. We further introduce SemComp-Bench, an evaluation protocol that uses a vision-language model (VLM) to answer structured binary questions. SemComp-Bench reports the OA Score and the GR Score for Outcome Achievement and Generation Reliability, respectively. Experiments on representative video generation models show that achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.
Sam: That's a good way to put it. And once it finds those two anchor points, it runs a consistency check to make sure the outcome frame is unambiguous — not blurry, not mid-transition.
Alex: Then what?
Sam: The third stage clips a short segment of video centred on that confirmed outcome — at least three seconds long — by merging visually consistent neighbouring frames around it. And the fourth stage pairs each clip with instructions at two levels: a brief description and a detailed version that specifies exact actions, like "tie a knot at the top left corner."
Alex: Why bother with two levels of instruction?
Sam: Because the level of detail turns out to matter quite a bit for how well the models perform. Detailed instructions help models better define what the finished result should look like — but they also demand much more from the AI's ability to understand meaning. Brief instructions are easier to follow in one sense, but leave more room for the model to go off in the wrong direction.
Alex: So how do they actually score the models on these tasks?
Sam: They use a vision-language model to ask structured yes-or-no questions about each generated video. Things like: does the finished object match what was asked for? Does the visual style stay consistent with the reference image? Is there any sudden jump or discontinuity that suggests the model lost track of what it was doing?
Alex: And the model has to pass all of those checks together?
Sam: Right — it's not enough to score well on one criterion and badly on another. The overall outcome score only counts if the video clears every validity check. And here's where the results get sobering: the best models tested still scored below forty percent on that joint measure.
Alex: That's quite low. What's going wrong?
Sam: The experiments point to two recurring failures. First, models can generate convincing individual frames, but they struggle to maintain a stable, logical visual progression between frames — so the transformation doesn't hold together as a coherent sequence. Second, they lose track of the reference image's specific details during complex changes. The leather stops looking like that leather. The object drifts.
Alex: It's like a student who can draw a good first sketch and a good final sketch, but can't connect them into a believable process.
Sam: That's a fair analogy. And it highlights why this benchmark is measuring something genuinely different from what came before.
Alex: One thing I want to ask about — how reliable is the scoring system itself? If you run the same evaluation twice, do you get the same numbers?
Sam: The authors checked this by running the evaluation multiple times and measuring how much the scores varied. For most of the primary metrics, the variation was relatively small — the system is reasonably stable. But certain diagnostic checks showed more variability on specific models, which is worth noting.
Alex: And there's a deeper issue there, isn't there — the whole pipeline is built on an AI judging other AIs.
Sam: That's the central limitation the authors are candid about. If the vision-language model used for scoring has a blind spot — if it consistently misreads a particular kind of visual detail — then every score it produces inherits that bias. The authors flag this throughout their analysis rather than glossing over it.
Alex: So where does this leave the field?
Sam: The paper suggests that this kind of structured feedback — specific, binary, visually grounded — could be used to directly train future video models to get better at physical transformations and task completion. Rather than just rewarding videos that look smooth, you'd be rewarding videos that actually work.
Alex: That's a meaningful shift in how we think about measuring video AI — away from counting pretty pixels and toward verifying real-world outcomes.
Sam: That's the core of it. Thanks for listening to ResearchPod.