ReGap-Bench paper: matched visual references lift multimodal accuracy by up to 25%
Multimodal large language models can recognize what is in an image yet still struggle to reason over it reliably across several steps. The paper gives a counting example: a model may see all the objects but lose track of which ones it has already counted.
The authors argue that such failures, commonly blamed on perception, point to a different problem, which they call a reference gap: the failure to bind language to the relevant visual entities, regions, or locations. Motivated by the theory of visual routines, they test whether external references, such as numbering, can narrow that gap.
To do this they introduce ReGap-Bench, a paired benchmark of five visual reasoning tasks. In each pair the scene, the question and the answer are fixed, and only the reference varies. That design isolates the effect of the reference from everything else.
The reported result is that task-matched references improve accuracy by up to 25%, while mismatched references provide little benefit or even hurt performance. From this the authors conclude that the reference gap is a bottleneck distinct from visual perception, and that constructing task-appropriate visual references can serve as a training-free visual skill for improving multimodal reasoning.
Key facts
- The paper introduces ReGap-Bench, a paired benchmark of five visual reasoning tasks where the scene, question and answer stay fixed and only the reference varies.
- The authors define a 'reference gap': the failure to bind language to the relevant visual entities, regions, or locations.
- Task-matched references, such as numbering, improve accuracy by up to 25%.
- Mismatched references provide little benefit or even hurt performance.
- The authors say the gap is a bottleneck distinct from visual perception and that building task-appropriate references can work as a training-free visual skill.
Why it matters
Failures of multimodal models on tasks like counting are commonly put down to weak perception. This paper argues for a different diagnosis: the model sees the content but cannot keep track of which entities, regions or locations its words refer to. If that holds, the fix is not necessarily a better vision encoder. The authors present it as a bottleneck distinct from visual perception, motivated by the theory of visual routines.
Who it affects
Researchers who build and evaluate multimodal LLMs are the direct audience, since ReGap-Bench is a new way to separate reference problems from perception problems. Anyone who prompts such models on multi-step visual tasks such as counting may also care, because the paper suggests the way a scene is annotated with references changes the outcome.
How to use it
The practical idea is to add an external reference to the image, such as numbering the objects, and to match it to the task. The authors describe constructing task-appropriate visual references as a training-free visual skill, meaning no retraining is involved. The abstract's warning is that a reference that does not fit the task gives little benefit or can hurt, so the choice of reference matters as much as having one.
How solid is it
This is a preprint abstract-level account, listed on Hugging Face. The headline figure is that task-matched references improve accuracy by up to 25%; that is a ceiling, and the source does not say which of the five tasks produced it or whether it is percentage points or a relative gain. The benchmark design, with the scene, question and answer fixed and only the reference varying, is a controlled setup. The source does not name the models tested.
Risks and caveats
The 'up to 25%' figure is a best case, not an average. The source gives no size for the harm from mismatched references, does not list the five tasks, and does not name the models tested, so how far the result generalises cannot be judged from it. The claim that the reference gap is distinct from perception is the authors' interpretation of their results.
“In counting, a model may see all objects but lose track of which ones it has counted.”
— From the paper's abstract on ReGap-Bench