Glonce

AskStream benchmark tests whether video AI asks for missing information

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact40Novelty55Relevance100Surprise25Players10Credibility70
AskStream benchmark tests whether video AI asks for missing information

A preprint introduces AskStream, a benchmark for streaming video understanding that asks a question existing tests skip: when, and what, should a model ask before it answers?

The authors start from how video AI interaction has developed. Multimodal large language models moved from offline video understanding to streaming, and then to proactive interaction, where a model decides for itself when to answer instead of only responding on demand. The catch, they say, is that existing proactive paradigms typically assume the information needed to answer a query is complete. That leaves the model two options: wait or answer.

Real situations are messier. Information is often incomplete because of uncertainty, either in the video or in what the user said. The authors state that under such uncertainty a proactive model can wait forever or produce highly hallucinatory responses. Their proposal is to redefine proactive interaction with a third action, ask, so a model can actively request the missing information.

To study this, they built AskStream, which they describe as the first multi-turn benchmark spanning video and user uncertainty for streaming video understanding. It has four task categories and eight subtasks. It contains 438 videos with 2,714 manually annotated frame-level time windows, each labelled as one of three types: wait, ask, or answer.

The authors tested models in both multiple-choice and open-ended settings. Even the strongest model attains only 52.18% on their proposed Ask-F1 metric. They read this as showing substantial limitations in proactive information seeking under uncertainty.

Key facts

  • AskStream adds a third action, "ask", to the wait-or-answer choice used in existing proactive streaming video benchmarks.
  • The benchmark has 438 videos and 2,714 manually annotated frame-level time windows, labelled wait, ask or answer.
  • It is organised into four task categories and eight subtasks, and covers uncertainty from both the video and the user.
  • Across multiple-choice and open-ended settings, the strongest model reaches only 52.18% on the proposed Ask-F1 metric.
  • The authors say the benchmark is the first multi-turn one spanning video and user uncertainty for streaming video understanding.

Why it matters

Proactive video models are meant to decide when to speak. The authors argue that this breaks down when the information needed for an answer is incomplete, because the model can wait forever or hallucinate. AskStream turns that gap into something measurable by treating asking for missing information as its own action alongside waiting and answering.

Who it affects

Researchers building streaming and proactive video understanding models are the direct audience, since the benchmark gives them a test for behaviour that earlier setups did not cover. The framing also bears on any assistant that watches a live video stream and talks to a user, where either the footage or the user's request may be unclear.

How to use it

The abstract does not describe release details, so there is nothing concrete to run yet. What it offers is a way of thinking: evaluate a streaming video model on three behaviours (wait, ask, answer) rather than two, and score it with Ask-F1 across multiple-choice and open-ended settings.

How solid is it

This is a preprint abstract, and the claims are the authors' own. The numbers are specific: 438 videos, 2,714 manually annotated time windows, four task categories, eight subtasks. The 52.18% figure is the best result reported across both settings, not an average. The abstract does not name the strongest model, and it does not give the definition of Ask-F1 or the other models' scores.

Risks and caveats

The "first of its kind" label is the authors' claim. The statement that uncertainty can make models wait forever or hallucinate is also theirs, stated as a possibility. A score of 52.18% only means something relative to the metric's definition and to other models, and neither is given here. Treat the headline result as an early signal until the full paper is checked.