Glonce

VGI-bench finds top video model Seedance 2.0 hits only 51%

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact58Novelty72Relevance98Surprise62Players32Credibility87
VGI-bench finds top video model Seedance 2.0 hits only 51%

Video generation models are increasingly showing signs of zero-shot visual reasoning through the frames they generate, but researchers say reliably measuring that ability is hard. A good benchmark, they argue, needs inputs aligned with the visual priors of current video models, needs to check that a model follows a valid evolving process rather than just landing on a plausible final frame, and needs task difficulty calibrated to stay challenging yet still partly solvable.

To meet those requirements the researchers built VGI-bench, containing 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags for fine-grained evaluation of a video generation model's visual reasoning.

Running current systems through VGI-bench showed they can solve a subset of visually grounded reasoning tasks but remain far from reliable overall. The strongest model evaluated, Seedance 2.0, achieved only 51.0% under the benchmark's evaluation criteria.

The authors also dug into why. Their analysis covers the models' output failure modes, how sensitive results are to input conditions, and how far performance transfers from synthetic fine-tuning. Looking at the internal denoising process, they found limited self-correction: later denoising steps mostly refine the early hypothesis a model settles on rather than fix genuine reasoning errors made earlier in generation.

The authors say they hope VGI-bench will help push development of next-generation video generation models forward. The project has a public website at hexuan21.github.io/VGI-Bench.

Key facts

  • VGI-bench comprises 27 tasks and 810 instances, organized by a two-level taxonomy of task domains and skill tags.
  • The benchmark requires a valid evolving reasoning process, not just a plausible final frame, and calibrates task difficulty to stay challenging yet partly feasible.
  • The strongest model evaluated, Seedance 2.0, scored only 51.0% under the benchmark's evaluation criteria.
  • Current generative systems can solve a subset of visually grounded reasoning tasks but remain far from reliable overall.
  • Analysis of the internal denoising process found limited self-correction: later steps mainly refine early hypotheses rather than correct reasoning errors.

Why it matters

Video generation models are starting to show zero-shot visual reasoning through the frames they produce, but there has not been a benchmark built around how these models actually work. VGI-bench targets that gap by aligning its inputs with the visual priors of current video models and by checking the reasoning process itself, not just whether the final frame looks plausible. That distinction matters: a convincing final image can still sit on top of a broken reasoning path, and a benchmark that only judges endpoints would miss it.

Who it affects

Researchers and teams building or evaluating video generation models are the direct audience, including whoever develops systems like Seedance 2.0. Anyone relying on video generation models for tasks that require actual visual reasoning, rather than plain image synthesis, is also affected: the results show accuracy stays well short of reliable even for the best model tested.

How to use it

The project has a public website at hexuan21.github.io/VGI-Bench, though the source text does not describe a release date, license terms, or how the tasks and instances can be accessed. What it does give teams is a structured set of 27 tasks and 810 instances, organized by domain and skill tag, for checking a video generation model's visual reasoning instead of relying on ad hoc checks of final frames.

How solid is it

VGI-bench's 27 tasks and 810 instances were built specifically to require a valid evolving reasoning process rather than a merely plausible outcome, and to keep difficulty calibrated so tasks stay challenging yet partly solvable. The headline number backs that design up: the strongest model tested, Seedance 2.0, reached only 51.0% under the benchmark's evaluation criteria. The authors extended the analysis beyond that single score to output failure modes, sensitivity to input conditions, and how performance transfers from synthetic fine-tuning.

Risks and caveats

The source text does not name VGI-bench's authors or their institutions, give a release date or publication venue, or report scores for any model besides Seedance 2.0, so how other video generation models compare is not established here. It also does not spell out exactly what the 51.0% figure measures beyond 'under our evaluation criteria,' so the precise metric behind that number, whether accuracy, pass rate, or something else, is not stated.