New BVB benchmark tests video understanding via Blender rebuilds
Researchers have introduced BVB, short for Blender-VideoBench, a benchmark for testing whether AI agents genuinely understand video content rather than just answering questions about it. The idea: if an agent truly understands a video, it should be able to reconstruct it programmatically, by writing code that rebuilds the scene in 3D software, instead of relying on diffusion-based video generation.
Agents are given real-world videos and asked to reconstruct them as animated scenes in Blender, the open-source 3D tool. To keep the comparison fair, every agent works through a shared lightweight harness called Mini-BVB, inside an identical sandbox and under the same cost limit. The benchmark then renders each agent's reconstruction from its own animated camera and scores it on two axes. Dual VQA measures how many spatiotemporal facts, what happened, where, and when in the scene, the reconstruction preserves. Latent Similarity measures how closely the reconstruction matches the original video perceptually, meaning how similar it looks and feels. The two scores combine into an overall square-root-mean score that favors balanced performance across both axes.
The researchers evaluated 51 configurations drawn from 10 model families. The best-performing configuration reached a Latent Similarity score of 88.6, meaning its reconstructions looked and felt close to the source videos, but it retained only 53.7% of the source-correct spatiotemporal answers on Dual VQA. In other words, a reconstruction could look convincing while still getting a large share of the underlying facts wrong. Giving models additional reasoning effort improved visual similarity but did not close this gap in factual accuracy: models got better at looking right without getting proportionally better at being right.
To check whether Latent Similarity actually tracks what people care about, the researchers ran a blind study with 15 human raters comparing five configurations, and found that Latent Similarity correlates strongly with human preference. They conclude that programmatic reconstruction in Blender is a viable test of agentic video understanding, and that semantic retention, keeping the actual facts of a scene intact rather than just its look, remains the main unsolved challenge.
Key facts
- BVB (Blender-VideoBench) tests AI agents by having them reconstruct real videos as animated Blender scenes instead of answering questions about them.
- Reconstructions are scored on two axes: Dual VQA, which measures spatiotemporal fact retention, and Latent Similarity, which measures perceptual match to the source.
- Across 51 configurations from 10 model families, the best configuration scored 88.6 Latent Similarity but retained only 53.7% of source-correct spatiotemporal answers.
- Extra reasoning effort improved visual similarity but did not close the gap in factual accuracy.
- A blind study with 15 raters across five configurations found Latent Similarity correlates strongly with human preference.
Why it matters
Most video understanding benchmarks still test models through question answering, which can reward pattern matching over real comprehension. BVB instead asks agents to prove understanding by rebuilding a video from scratch as code, then measures both whether the facts of the scene survive and whether the rebuild looks right. The gap the researchers found, a model that looks convincing on 88.6 Latent Similarity while getting almost half its spatiotemporal facts wrong, shows that visual plausibility and genuine scene understanding are not the same thing, even as agentic coding tools get better at producing polished-looking output.
Who it affects
AI labs and researchers building or evaluating multimodal and agentic models for video understanding, and anyone assessing whether AI coding agents can be trusted to translate what they see into accurate structured output rather than just a plausible-looking one. It also matters to the benchmark community, since BVB proposes reconstruction as an alternative to question-answering style evaluation.
How to use it
BVB is a benchmark and evaluation methodology, not a product. Agents are run through the Mini-BVB harness inside an identical sandbox under a shared cost limit, which lets researchers compare models on equal footing and score new models on the same two axes, Dual VQA and Latent Similarity, that the paper defines.
How solid is it
The evaluation spans 51 configurations across 10 model families, giving reasonably broad coverage of current models. The researchers also validated their Latent Similarity metric against human judgment with a blind study of 15 raters across five configurations and found a strong correlation, which supports treating perceptual similarity as a meaningful proxy for human preference. The source does not disclose the identity of the best-performing model, name any of the 10 model families or 51 configurations individually, or give the full scoring formula for Dual VQA and the square-root-mean score beyond what is described above.
Risks and caveats
The central finding, that the best model's outputs look far more faithful than they are, is itself a caution against judging agentic video understanding by appearance alone; a reconstruction benchmark that leaned only on Latent Similarity could reward superficial visual mimicry. The paper's abstract gives no publication or submission date, and the human-preference check rests on a single blind study with 15 raters, a useful but modest sample.
“Additional reasoning improves visual similarity but does not close this gap in factual accuracy.”
— the paper's authors