Video-IFBench tests whether MLLMs actually follow instructions on video
Researchers introduced Video-IFBench, a benchmark built to evaluate instruction following in video understanding by multimodal large language models (MLLMs). The gap it targets: existing video benchmarks mostly score whether a model gets the content right, not whether it obeys the specific constraints a user attaches to a request, so this capability has gone largely untested. Video-IFBench evaluates models on their ability to satisfy diverse user-specified constraints, including ones grounded in what is seen (visual content) and what is heard (audio content) in the video, not just the visible content alone. The benchmark's taxonomy covers four instruction templates (single-task, multi-task, selection, and nested instructions), spanning 32 task types and 39 manually designed constraint categories that mix semantic requirements (what the answer must say) with format requirements (how the answer must be structured). To keep annotation practical at that scale, the team built a semi-automatic data construction pipeline combining MLLMs, programmatic processing, and human verification, producing 1.5K samples. Running more than 20 recent MLLMs through the benchmark, the authors found that instruction following in video understanding remains difficult for current models, and gets markedly harder on instructions carrying many constraints, semantic constraints, or complex conditional structures where the model has to pick the correct branch or path depending on what happens in the video. The paper does not report individual model scores, name which models were tested, or state an institution, publication venue, or release date; those details are absent from the available text.
Key facts
- Video-IFBench evaluates whether MLLMs follow user-specified constraints on video tasks, not just whether they describe video content accurately.
- The taxonomy has four instruction templates (single-task, multi-task, selection, nested) covering 32 task types and 39 manually designed constraint categories spanning semantic and format requirements.
- A semi-automatic pipeline combining MLLMs, programmatic processing, and human verification produced 1.5K samples for the benchmark.
- More than 20 recent MLLMs were evaluated; the authors report that instruction following remains challenging, especially with many constraints, semantic constraints, or complex conditional branching based on video content.
- No per-model scores, model names, institution, or publication venue are given in the available text.
Why it matters
Most video benchmarks for MLLMs check whether a model understood what happened on screen. Video-IFBench checks a different, more practical thing: whether a model does what it was actually asked to do, including constraints tied to both video and audio content. That distinction matters because a model can describe a video correctly while still ignoring the format, scope, or conditional logic a user specified, and that failure mode had gone largely unmeasured.
Who it affects
The benchmark is aimed at researchers and teams building or evaluating multimodal LLMs for video tasks, since it gives them a way to test instruction adherence separately from raw content-understanding accuracy. The underlying finding, that current MLLMs struggle with constrained and conditional video instructions, is relevant to anyone building products that ask a video model to follow specific rules rather than produce free-form descriptions.
How to use it
Video-IFBench is presented as an evaluation benchmark: 1.5K samples across 32 task types and 39 constraint categories, built through a semi-automatic pipeline mixing model-generated data, programmatic processing, and human verification. The available text does not state a release venue, license, or availability details for the dataset or code.
How solid is it
The evaluation is large-scale by the field's standards, covering more than 20 recent MLLMs, and the constraint taxonomy was manually designed rather than left to automatic generation alone, with human verification built into the data pipeline. The source text does not give individual model scores or name the models tested, so the specific magnitude of the difficulty gap across models cannot be assessed from what is available.
Risks and caveats
The claim that current models struggle most on instructions with many constraints, semantic constraints, or complex conditional branching comes directly from the authors and is not independently broken down by number here, since no per-model or per-category scores appear in the source. No institution, publication venue, or release date is stated, which limits independent verification of the work's provenance.