VibeVideoBench tests AI agents that write code to make videos
A new paper introduces VibeVideoBench, a benchmark for agents that make videos by writing code instead of using video generation models. The authors start from a limit of diffusion video models: they produce striking clips but cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. The alternative they point to is multimodal agents that write code to render every frame and synthesize every sound.
The benchmark has 239 self-contained tasks in English and Chinese, spread over 12 categories that run from data stories and music-driven pieces to classroom lessons and paper explainers. Each task asks an agent to deliver a finished MP4 with sound within one hour. The agent runs in its native harness inside a sandbox that has no image or video generation model, so everything on screen and in the audio has to come from code the agent writes.
Scoring is hierarchical and evidence-based. It combines program checks of the delivery spec, about 13k task-specific rubric checks traced to the requirements they enforce, and five quality dimensions anchored per task and scored by three VLM judge panels.
The authors ran nine frontier agent systems, with 2,151 scored rollouts in total. GPT-6 Astra and Claude Opus 5.5 rank first and second, both with the judges and with human annotators. The strongest system satisfies every core requirement on 76.3% of tasks.
The paper treats delivery as a second axis of evaluation. Counting missing videos as zero changes the ranking. Three quarters of the lost videos are final renders that started too late, ran too slowly or never started, which the authors call a failure of how agents plan their hour. Agents build their own rendering pipelines and inspect their own frames, yet how much they look does not predict quality. Finally, the judges' ranking of the systems closely matches blind human preferences.
Key facts
- VibeVideoBench has 239 self-contained tasks in English and Chinese across 12 categories, from data stories and music-driven pieces to classroom lessons and paper explainers.
- Each task requires a finished MP4 with sound within one hour, produced in a sandbox with no image or video generation model.
- Across nine frontier agent systems and 2,151 scored rollouts, GPT-6 Astra and Claude Opus 5.5 rank first and second with both the judges and human annotators.
- The strongest system satisfies every core requirement on 76.3% of tasks.
- Counting missing videos as zero changes the ranking; three quarters of the lost videos are final renders that started too late, ran too slowly or never started.
Why it matters
Diffusion video models make striking clips, but the authors say they cannot reliably render an exact formula, a correct chart, a rule-consistent game or narration synchronized to on-screen events. The paper bets on a different route: agents that write code to render every frame and synthesize every sound. VibeVideoBench gives that route a shared test, with tasks such as data stories, classroom lessons and paper explainers where correctness matters as much as looks.
Who it affects
Teams building coding and multimodal agents are the main audience, since the benchmark measures how well an agent handles a long, open-ended build under a one-hour limit. It also concerns anyone who wants videos with exact charts, formulas or synchronized narration, the cases where the authors say diffusion models fall short.
How to use it
The benchmark can serve as a test of an agent in its native harness: give it a task, a sandbox without image or video generation models and one hour, then check the delivered MP4 against the spec, the task-specific rubric checks and the quality dimensions. The text does not state a release date, code link or dataset availability. The results also hint at a practical lesson: since most lost videos came from final renders that started late or never started, planning the hour matters.
How solid is it
The evaluation is broad: nine frontier agent systems, 2,151 scored rollouts, about 13k rubric checks and three VLM judge panels. The authors report that the judges' ranking closely matches blind human preferences, and that GPT-6 Astra and Claude Opus 5.5 rank first and second with both the judges and human annotators. All of this is the authors' own account from the abstract. The abstract names no authors or institutions, and it does not name the other seven evaluated systems, the three VLM judge models or the five quality dimensions.
Risks and caveats
The 76.3% figure belongs to the strongest system only, and the abstract does not say which system that is; no per-system scores are given. The size of the ranking change when missing videos count as zero is not stated. The one-hour limit and the sandbox without image or video generation models define a specific setting, so results say little about other setups. The authors also note that how much agents inspect their own frames does not predict quality.
“Delivery is a second axis: counting missing videos as zero changes the ranking, and three quarters of the lost videos are final renders that started too late, ran too slowly or never started, a failure of how agents plan their hour.”
— VibeVideoBench paper abstract