Claude 4.6 beats GPT 5.4 on new relational reasoning benchmark
A new paper introduces SciReC, a model-adaptive multimodal academic dialog benchmark built to test relational reasoning in multimodal large language models (MLLMs), the ability to perceive, compare and integrate relationships between concepts. The benchmark covers multiple categories of this ability, including analogical, structural and cause-effect relations, each capturing a different aspect of higher-order understanding. Alongside the benchmark, the authors propose DMRA, a deficit-based diagnostic framework that quantifies how much three underlying factors, visual understanding, exhibiting knowledge, and memory recall, each contribute when a model fails on a task, in order to identify the primary cause of the failure. On the overall relational score, Claude 4.6 came out on top with 73%, followed by GPT 5.4 at 68%. Beyond the headline scores, the results split by model type and by subject domain. Open-source models posted their lowest scores on spatial relations, while proprietary models struggled more with hierarchical and sequential relations. Across academic domains, every model performed worst on Astronomy questions and best on Psychology questions. Running DMRA's diagnosis across all models, relational reasoning itself was the leading source of error, with memory limitations the second most common cause. The text does not name the paper's authors or institutions, does not state how many models were evaluated or how large the underlying dataset is, does not name which specific open-source or proprietary models were tested beyond those two category labels, and does not explain the mechanics of how DMRA attributes an error to visual understanding, exhibiting knowledge or memory recall, only that it does.
Key facts
- SciReC is a new model-adaptive multimodal academic dialog benchmark for relational reasoning, covering categories such as analogical, structural and cause-effect relations
- Claude 4.6 scored highest on the overall relational score at 73%, with GPT 5.4 second at 68%
- Open-source models scored lowest on spatial relations; proprietary models struggled more with hierarchical and sequential relations
- Across academic domains, every model performed worst on Astronomy and best on Psychology
- The paper's DMRA diagnostic framework found relational reasoning itself is the leading cause of model errors, with memory limitations second
Why it matters
Most multimodal benchmarks score models on whether an answer is right, not on why a model got it wrong. SciReC pairs a relational reasoning benchmark with DMRA, a diagnostic layer that breaks a failure down into how much visual understanding, exhibiting knowledge and memory recall each contributed, so a low score comes with a stated cause rather than just a number.
Who it affects
The results speak most directly to teams building or evaluating multimodal models: Claude 4.6 leads the overall relational score, GPT 5.4 follows, and the split between open-source and proprietary models on spatial versus hierarchical and sequential relations gives both camps a concrete weak spot to target. Anyone using these models for tasks that lean on academic or diagrammatic reasoning, such as science tutoring or research assistance, is affected by the domain gap: every model tested did worst on Astronomy and best on Psychology.
How to use it
The paper does not state pricing, licensing or access terms for SciReC or DMRA, and the source text names no specific models beyond the 'open-source' and 'proprietary' categories, so there is nothing further to extract on how to run or license the benchmark.
How solid is it
The source is an arXiv abstract rather than the full paper: it states the top-line scores, the category-level and domain-level trends, and the DMRA error breakdown, but it does not give the number of models evaluated, dataset size, publication date, author names or institutional affiliations, and it does not explain the mechanics behind DMRA's error attribution.
Risks and caveats
With only two models named and no count of how many systems or questions the benchmark actually covers, the reported percentages and trends should be read as this paper's specific findings rather than settled, field-wide conclusions about which model architecture is generally stronger at relational reasoning.