TestHallVQA benchmark exposes LVLM reasoning gaps under document redundancy

A group of researchers presents TestHallVQA, a benchmark for testing how large vision-language models (LVLMs) answer questions over multi-image, document-style inputs. The authors argue that existing planar VQA benchmarks typically isolate the challenge: some stress long-document understanding but with limited reasoning depth, while others demand complex visual reasoning but stay restricted to single-page, noise-free settings. No existing benchmark, in their account, combines both at once. Through theoretical analysis, the authors also identify a separate problem: irrelevant visual tokens measurably degrade an LVLM's performance, an effect they say has received little systematic quantification in prior work. TestHallVQA is built to address both gaps at the same time. It is a multi-image benchmark that combines document-level scale with the difficulty of human examinations, while covering a comprehensive range of tasks, and it lets researchers controllably inject multiple levels of contextual redundancy into the input. Building on that control, the authors propose a new evaluation metric, F1-R2, which jointly measures a model's computational reasoning capability and its robustness at retrieving the right evidence when the document is padded with redundant, irrelevant material. The abstract states that extensive experiments and analyses on mainstream LVLMs uncovered latent deficiencies across multiple dimensions, which the authors offer as concrete directions for future research. The dataset, the code, and the complete theoretical derivations are released publicly on GitHub.

Key facts

TestHallVQA is a multi-image VQA benchmark combining document-level scale with the difficulty of human exam questions and comprehensive task coverage.
The authors say prior planar VQA benchmarks isolate the challenge: either long-document scope with shallow reasoning, or complex reasoning confined to single clean pages.
A theoretical analysis identifies irrelevant visual tokens as a measurable, previously under-quantified source of performance degradation in LVLMs.
The paper proposes F1-R2, a metric that jointly scores reasoning capability and robustness to document-level redundancy, enabled by TestHallVQA's controllable redundancy injection.
Experiments on mainstream LVLMs reportedly surface deficiencies across multiple dimensions; the dataset, code and theoretical derivations are public on GitHub.

Why it matters

Document-style visual question answering keeps growing as a use case, from reading multi-page reports to scanned forms, but the authors argue the benchmarks used to measure it have a structural gap: they either test long documents with shallow reasoning demands, or they test hard reasoning on a single clean page, never both together. TestHallVQA is built to close that gap by combining document scale with exam-level reasoning difficulty in one benchmark. The paper also flags a subtler issue: LVLMs lose measurable accuracy when fed irrelevant visual tokens, a failure mode the authors say has not been systematically quantified before, which is what the new F1-R2 metric is designed to capture.

Who it affects

The benchmark targets researchers and engineering teams building or evaluating large vision-language models on document-style, multi-image reasoning tasks, including anyone benchmarking robustness to noisy or padded inputs. It is also relevant to teams deploying LVLMs on real multi-page documents, such as exam papers, scanned reports or forms, where irrelevant or redundant pages are common and where the paper's findings suggest current models may quietly lose accuracy.

How to use it

The dataset, the code and the complete theoretical derivations behind TestHallVQA and the F1-R2 metric are released publicly on GitHub at github.com/yqyu2317/TestHallVQA-benchmark. The benchmark supports controllably injecting multiple levels of contextual redundancy into its multi-image inputs, which is what lets researchers measure a model's reasoning and evidence-retrieval robustness separately rather than as one blended score.

How solid is it

This is an arXiv preprint, and the abstract available here reports the motivation, the benchmark design and the metric, but does not include the results themselves: no numeric results are reported, no accuracy scores, no F1-R2 values, and no benchmark table appear in the source text. It also does not name which mainstream LVLMs were evaluated, or describe TestHallVQA's size in terms of images, questions or exam subjects. The claim that the experiments reveal deficiencies across multiple dimensions is the authors' own characterization pending a look at the full paper's results section.

Risks and caveats

No author names, institutions or affiliations are given in the source text, and no submission or publication date appears either, so provenance cannot be verified from this abstract alone. Because no result figures are available here, readers cannot yet judge how large or how consistent the reported deficiencies are across the models tested, or which specific models were included.