OmniDR-Bench tests AI agents on video-centered deep research reports
The authors of a new preprint argue that deep research agents, which pull open-web evidence together into long reports, have no benchmark for research that starts from a source video. They introduce a task to fill that gap: Video-Centered Deep Research Report Generation. Given a video and a user request, an agent identifies research questions, retrieves external multimodal evidence, and produces an interactive report linked to relevant video moments. The authors say the task has direct practical value because it helps users locate relevant moments, verify claims, reproduce demonstrated procedures, and make informed decisions from video content.
The paper presents three pieces. The first is a skill-based workflow for the task. The second is OmniDR-Bench, a benchmark of 50 Chinese and English tasks across 19 real-world subcategories, with 6,414 criteria in total. The third is the evaluation scheme, Omni Atomic Rubric Trees (Omni-ARTs). Each tree contains three dimensions, 23 shared directions, and task-specific atomic binary criteria. The 23 directions test audiovisual grounding, multimodal evidence use, and interactive presentation.
The rubric trees are built with a process called Audited Rubric Construction (ARC), which runs through four steps: independent evidence collection, automated checks, behavioral calibration, and iterative human review.
On the benchmark, 18 agent configurations were evaluated. The best system scores 63.3 out of 100. The authors conclude that current agents still need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support. They also analyze models, harnesses, skill configurations, video categories, and research trajectories. The code and data are released on GitHub at marinero4972/OmniDR-Bench.
Key facts
- The paper defines Video-Centered Deep Research Report Generation: from a video and a user request, an agent produces an interactive report linked to relevant video moments, using external multimodal evidence.
- OmniDR-Bench has 50 Chinese and English tasks across 19 real-world subcategories and 6,414 criteria.
- Evaluation uses Omni Atomic Rubric Trees: three dimensions, 23 shared directions, and task-specific atomic binary criteria, built through Audited Rubric Construction.
- Across 18 agent configurations, the best system scores 63.3 out of 100.
- The authors say agents need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support; code and data are released.
Why it matters
The authors say current benchmarks do not evaluate research centered on a source video. Deep research agents are usually tested on open-web evidence and long text reports. This work adds a task where the video is the starting point and the output is a report tied to specific moments in it. That makes grounding in audio and visuals, not just in web text, part of what gets scored.
Who it affects
Researchers building and evaluating deep research agents and multimodal systems are the direct audience. The authors also point to practical users of video content: people who want to locate relevant moments, verify claims, reproduce demonstrated procedures, or make decisions based on a video. The benchmark covers both Chinese and English tasks.
How to use it
The code and data are released at https://github.com/marinero4972/OmniDR-Bench. Teams can run their own agents on the 50 tasks and score them with the Omni-ARTs rubric trees, which cover audiovisual grounding, multimodal evidence use, and interactive presentation. The authors also analyze models, harnesses, skill configurations, video categories, and research trajectories, which can guide comparisons.
How solid is it
This is a preprint, and the results are the authors' own. The headline number is a maximum: the best of 18 agent configurations scores 63.3 out of 100. The rubrics were built through independent evidence collection, automated checks, behavioral calibration, and iterative human review. The abstract does not name the top system, give other systems' scores, or say how the 100-point score is computed from the criteria. It also reports no human baseline.
Risks and caveats
The benchmark is small: 50 tasks, though they carry 6,414 criteria. The source does not say how many tasks are Chinese versus English or how many videos are used. Because the abstract does not say which configuration scored 63.3 or how the score is aggregated, the figure is hard to interpret beyond showing there is room for improvement. Findings about agent weaknesses are the authors' own conclusions.
“Current agents still need broader content coverage, stronger causal and logical analysis across modalities, and more accurate citation support.”
— From the paper's abstract