SemComp-Bench grades video generation by outcome, not appearance
Researchers introduce Semantic Task Completion Video Generation, a new way of framing video generation as outcome oriented rather than appearance oriented. Under this formulation, a generated video only counts as successful if it clears two separate bars: it has to achieve the intended outcome, and it has to stay semantically grounded in the reference image, meaning the high level meaning of the generated outcome has to correspond to what the reference image was asking for. Evaluation is aimed squarely at the outcome shown in the finished clip. It does not require the video to walk through a complete sequence of intermediate steps, and it does not require the close visual matching to the reference image that conventional appearance consistency checks look for.
To evaluate the idea systematically, the researchers built SemComp-Data, an evaluation dataset spanning six domains. Every instance in the dataset pairs a reference image with a detailed instruction, a brief instruction, and an outcome-centric video clip. The dataset is assembled through a scalable four-stage curation pipeline that converts raw video footage into these standardized instances.
On top of the dataset, the researchers introduce SemComp-Bench, an evaluation protocol that hands a generated video to a vision-language model and has it answer a set of structured binary questions about that clip. The protocol turns those answers into two separate scores: an OA Score for Outcome Achievement, measuring whether the intended result actually happened, and a GR Score for Generation Reliability.
Running the benchmark on representative video generation models shows that hitting both requirements at once, achieving the intended outcome and keeping the result semantically grounded in the reference image, remains difficult for current systems. The abstract describes this only qualitatively, as 'challenging,' without giving the actual OA Score or GR Score numbers behind the finding.
Key facts
- Introduces Semantic Task Completion Video Generation, a task formulation where success requires both achieving the intended outcome and staying semantically grounded in the reference image.
- The evaluation deliberately drops two older requirements: showing a complete sequence of intermediate task steps, and matching the reference image's appearance in the conventional sense.
- SemComp-Data, the companion dataset, spans six domains; each instance pairs a reference image, a detailed instruction, a brief instruction, and an outcome-centric video clip, built via a scalable four-stage curation pipeline.
- SemComp-Bench, the evaluation protocol, uses a vision-language model to answer structured binary questions and reports two separate scores, an OA Score for outcome achievement and a GR Score for generation reliability.
- Tests on representative video generation models found that achieving the intended outcome while keeping semantic grounding in the reference image remains challenging, though no numeric scores are given in the abstract.
Why it matters
Most existing ways of scoring video generation models still lean on whether a generated clip looks like its reference frame or walks through a plausible sequence of steps. This paper argues that is the wrong test for videos meant to show a task getting done: a clip can look right and still fail to depict the outcome actually happening. The Semantic Task Completion Video Generation formulation splits success into two separate requirements, achieving the intended outcome and staying semantically grounded in the reference image's high-level meaning, and explicitly drops the older requirements of full step-by-step sequences and strict appearance matching. That reframing, backed by the SemComp-Data dataset and the SemComp-Bench scoring protocol built to measure it, is the paper's contribution.
Who it affects
Anyone building or evaluating video generation models meant to show a task being completed, rather than purely decorative or stylistic clips, gets a new way to check whether a model's output is actually correct rather than just visually convincing. The abstract does not name specific companies, products, or research groups as adopters, and it does not identify which video generation models were tested beyond calling them 'representative.'
How to use it
SemComp-Bench works by having a vision-language model answer a set of structured binary questions about each generated video clip, then rolling the answers into two scores: an OA Score for whether the intended outcome was reached, and a GR Score for how reliably the generation held together. A model evaluated against SemComp-Data would be scored the same way, using the reference image, the detailed and brief instructions, and the outcome-centric clip that make up each of the dataset's instances. The abstract does not say whether the dataset, the benchmark's question set, or any code are being released for other researchers to run.
How solid is it
The claims rest on a single paper abstract, submitted to Hugging Face's papers listing under the name Keyu Tu, with no institutional affiliation attached, and drawing light community attention so far, 20 upvotes and one comment. The central finding is qualitative rather than quantitative: the abstract states that achieving intended outcomes while keeping semantic grounding 'remains challenging' for the video generation models tested, without giving the actual OA Score or GR Score numbers behind that claim.
Risks and caveats
The abstract does not name the six domains SemComp-Data covers, does not identify which video generation models were treated as 'representative' in the reported experiments, does not state how many instances the dataset contains, and gives no comparison against any earlier evaluation protocol or benchmark. It also does not say whether the dataset, the benchmark's question set, or any code are being made available to other researchers.
“Achieving intended outcomes while maintaining task-relevant semantic grounding in reference images remains challenging.”
— the authors