Glonce

VideoMemBench: Claude Opus 5 tops video-memory test at only 45.7%

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact42Novelty60Relevance96Surprise35Players25Credibility72
VideoMemBench: Claude Opus 5 tops video-memory test at only 45.7%

Multimodal large language models can now process hour-long videos, which makes a long, multi-turn conversation about a single video a natural way to use them. The paper behind VideoMemBench argues that sustaining such a conversation takes more than answering each question on its own. The model has to retrieve and reuse what was established earlier, both what it has seen and what it has said. The authors call this ability in-context video memory.

They say existing video QA benchmarks leave it unmeasured. Their questions are mutually independent, never reference one another, and each can be answered from the video alone, so nothing forces a model to carry information from one turn to the next.

In VideoMemBench, each sample pairs one video with a long multi-turn dialogue. The turns are linked by typed cross-turn dependencies. An Episodic Dependency refers back to a visual episode described in an earlier turn, and so probes memory of what the model has seen. An Answer Dependency builds on the model's own earlier answer, and so probes memory of what the model has said.

Every dependency is checked for two properties. Necessity means the question cannot be answered without its earlier turns. Non-leakage means the question does not restate the information from those earlier turns. To enforce both, the authors built a construction pipeline with three LLM roles: a planner, a generator and a judger. Human experts review each dialogue.

The benchmark has 326 videos and 2,476 turns. Of those turns, 1,387 depend on earlier turns, and the dependencies include multi-parent dependencies and long-range call-back. Over 20 open- and closed-source MLLMs were evaluated. The best performer, Claude Opus 5, reaches an overall score of only 45.7%, which the authors take as evidence of how hard in-context video memory is. The dataset and evaluation are released.

Key facts

  • VideoMemBench pairs each of 326 videos with a long multi-turn dialogue, 2,476 turns in total, of which 1,387 depend on earlier turns.
  • Two typed dependencies: Episodic (refers back to a visual episode from an earlier turn) and Answer (builds on the model's own earlier answer).
  • Every dependency is verified for necessity and non-leakage; a planner, a generator and a judger LLM build the dialogues, and human experts review each one.
  • Among over 20 open- and closed-source MLLMs, Claude Opus 5 is best with an overall score of only 45.7%.
  • The dataset and evaluation are released.

Why it matters

Video models are marketed on how long a video they can take in, but real use is often a conversation: ask, follow up, refer back to something said or shown ten turns ago. The authors say existing video QA benchmarks never test this, because their questions are independent and each can be answered from the video alone. VideoMemBench is built to close that gap by making later questions depend on earlier turns. A top score of 45.7% from the best of over 20 models suggests the gap is a real one.

Who it affects

Researchers building and evaluating multimodal LLMs get a benchmark aimed at a skill that single-question video QA does not measure. Teams building products around long video conversations, where a user keeps questioning one video, are the natural readers of the results, since the benchmark targets exactly that mode of use.

How to use it

The authors state that the dataset and evaluation are released. A model can be run on the dialogues turn by turn, with the benchmark's evaluation producing an overall score, the figure the paper reports for Claude Opus 5. The abstract gives no download link or licence, so check the paper page for those details.

How solid is it

The evidence is the authors' own abstract, so the numbers come from the paper itself. The construction method is described concretely: dependencies are checked for necessity and non-leakage, three LLM roles generate the dialogues, and human experts review each one. Scores are named only for the top model, Claude Opus 5 at 45.7% overall. The abstract does not list the other evaluated models or report a human baseline, so the 45.7% has no stated reference point.

Risks and caveats

Because the abstract gives only one score, it is hard to tell how large the spread between models is or how the score splits between Episodic and Answer Dependency turns. The dialogues are built with LLMs in the planner, generator and judger roles, so the data inherits whatever those models bring, though human review is meant to catch problems. Only 1,387 of the 2,476 turns depend on earlier turns, so the memory signal comes from just over half the benchmark. A low score also says the task is hard, not why models fail.

“Existing video QA benchmarks leave this ability unmeasured”

— VideoMemBench paper abstract