OmniAssistBench reveals Omni-LLMs struggle as video assistants

Researchers introduced OmniAssistBench, a benchmark for testing omni-modal large language models (Omni-LLMs) in the role of real-time video assistants: systems that continuously watch a scene and actively guide a user toward a goal, rather than passively describing a video after the fact. The authors argue that an interactive assistant has to combine what it currently sees, the user's goal, and prior knowledge to give useful help in the moment, and that evaluating this is unusually hard because a model's own unpredictable replies change what the user does next, something a static, prerecorded test set cannot capture.

The same user goal can often be reached by several different valid paths, which would make scoring inconsistent, so the authors give each model predefined priors drawn from the source video and require it to guide the user along that exact route rather than any workable alternative. Because genuine interactive assistant footage is rare, they built the dataset by reverse engineering existing internet videos: deducing a plausible user goal behind each clip, then segmenting the footage into multi-turn clips that simulate a continuous back and forth between assistant and user. Assembling the dataset this way took over 1000 expert person-hours. The source names Xianyun Sun as author but no institution behind the project and gives no publication venue or date; it also does not state how many videos or clips the finished dataset contains, only the hours spent building it.

On the finished benchmark, the proprietary Gemini-3-Pro reaches a score of 66.4 out of a maximum of 100, and the open source Qwen3-Omni-Instruct reaches 51.2; the source does not say whether other models were tested. Both models generally understand what the user is asking, but they frequently answer incorrectly or incompletely. Specifically, they struggle with visual prompts such as hand gestures, they fail to maintain historical context across a multi-turn interaction, and they fail to delay their response until the target event has actually occurred. The authors conclude that the results point to substantial room for improvement before such models can become reliable assistants; the source gives no human baseline score to compare the 66.4 and 51.2 results against.

Key facts

OmniAssistBench tests Omni-LLMs (omni-modal large language models) as real-time video assistants that must actively guide a user toward a goal, not just describe a video after the fact.
The dataset was built by reverse engineering existing internet videos into multi-turn clips with predefined priors that force a model to guide the user along the exact route taken in the source video; assembling it took over 1000 expert person-hours.
The proprietary Gemini-3-Pro scores 66.4 out of a maximum of 100 on the benchmark, while the open source Qwen3-Omni-Instruct scores 51.2.
Both models generally understand what the user is asking but frequently answer incorrectly or incompletely.
Models struggle to read visual prompts such as hand gestures, fail to keep historical context across multi-turn interactions, and fail to delay their response until the target event occurs.

Why it matters

Passive video understanding, where a model answers questions about a clip it has already seen in full, is a largely solved evaluation problem; guiding a person through a task in real time is not, because the assistant's own replies change what the user does next, and a fixed, prerecorded test set cannot follow that back and forth. OmniAssistBench targets exactly that gap: it evaluates Omni-LLMs on their ability to actively combine what they currently see, a user's goal, and prior knowledge to give useful help step by step, using video reconstructed from the internet rather than a scripted lab recording. As omni-modal models move toward products that watch a live camera feed and coach a person through a task, a benchmark built around interactive guidance rather than static description addresses a real hole in how such systems get tested before they ship.

Who it affects

The results speak most directly to teams building or evaluating omni-modal models meant to act as live video assistants: the benchmark covers both a proprietary model, Gemini-3-Pro, and an open source one, Qwen3-Omni-Instruct, so the weaknesses cannot be dismissed as specific to one type of model. Companies planning to ship a camera-based assistant, such as an app that watches a user's hands and talks them through a physical task, are the most exposed, since the benchmark's named failure modes, misreading gestures, losing track of earlier turns, and answering too early, are exactly the behaviors such a product depends on getting right. End users of any such assistant are affected indirectly: today's models leave substantial room for error in the middle of a live, hands on task.

How to use it

For a team choosing or fine tuning an omni-modal model for a real-time assistant product, the benchmark's three named failure modes work as a pre-launch checklist rather than an abstract score: does the candidate model correctly read non-verbal cues like hand gestures, does it retain what happened earlier in a multi-turn session, and does it wait for the right moment, the target event, before it speaks instead of answering prematurely. The two published reference points, Gemini-3-Pro at 66.4 and Qwen3-Omni-Instruct at 51.2 out of 100, give a rough sense of where a proprietary and an open source model currently sit, though the source does not say whether any other models were evaluated, so the field of comparison beyond these two is unknown.

How solid is it

The dataset was not scripted for the benchmark; it was built by reverse engineering real internet videos, inferring a plausible user goal behind each one and cutting it into multi-turn clips that simulate a live back and forth, with models given predefined priors so they have to follow the exact route taken in the source footage rather than any workable alternative. That pipeline took over 1000 expert person-hours, which points to a labor intensive, carefully constructed dataset rather than an automated one. Against that, the source leaves several evaluation questions open: it names Xianyun Sun as author but no institution behind the project and gives no publication venue or date, it does not state how many videos or clips the finished dataset contains, and it gives no human baseline score against which to judge whether 66.4 or 51.2 out of 100 counts as good, mediocre, or poor performance in absolute terms.

Risks and caveats

The headline scores, 66.4 for the proprietary model and 51.2 for the open source one, are both well short of the maximum of 100. Both scores fall well short of reliability. The source is specific about where the points are lost: models struggle with visual prompts such as hand gestures, fail to maintain historical context across multi-turn interactions, and fail to delay their response until the target event has actually happened. That last failure, answering before the moment actually calls for it, is a particular risk for any product acting on a live camera feed, since a premature or context blind instruction during a physical task can send a user down the wrong step. The source reports results for only these two models and gives no indication of how many others were tested or excluded, so the comparison should be read as a snapshot of two systems rather than a survey of the field.