GPT and Claude reviewers draw about half their paper criticism from the appendix

Reviewers increasingly use large language models to point out weaknesses in papers, and the models' output often shapes the reviews humans submit. Earlier work compared how LLM reviews line up with human ones. What it left open is which parts of a paper the models actually criticize. A new preprint, listed on Hugging Face Papers, tackles that question.
The authors investigate 2,000 papers from four venues: ACL, EMNLP, COLM and ICLR. They report that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix. The source does not name specific model versions. The behaviour holds across three ways of using an LLM to review a paper: the API, a coding agent, and web access.
Human reviewers look very different. They draw only 8.4% of their weaknesses from the appendix, and those weaknesses mostly concern a paper's presentation and content tied to its core idea. The LLMs, by contrast, pull weaknesses from deep in the appendix. These concern fine-grained technical details that are not central to the paper's contributions.
The authors say they expect the analysis to contribute to the scholarly discourse on AI reviews. Their code is publicly available on GitHub at cincynlp/review_appendix.
Key facts
- The study covers 2,000 papers from ACL, EMNLP, COLM and ICLR.
- The latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix.
- Human reviewers draw only 8.4% of their weaknesses from the appendix, mostly on presentation and core-idea content.
- The appendix-heavy pattern holds across API, coding-agent and web access ways of running an LLM review.
- Code is public at github.com/cincynlp/review_appendix.
Why it matters
LLM-written criticism increasingly feeds into the reviews humans submit, so where the models look affects what authors get told. The study gives a concrete measure of that: about half or more of the models' listed weaknesses come from the appendix, against 8.4% for humans. The gap in focus is large, and it goes beyond how closely the two sets of reviews agree, which is what earlier work examined.
Who it affects
Authors submitting to venues like ACL, EMNLP, COLM and ICLR, and reviewers who lean on LLM output when writing their own reviews. It also concerns anyone studying how AI reviewing compares with human reviewing.
How to use it
The authors released their code at github.com/cincynlp/review_appendix, so others can examine the same appendix-versus-main-text pattern. The source gives no practical recommendations for reviewers or conferences.
How solid is it
The sample is sizeable at 2,000 papers from four venues, and the authors report the pattern across three access methods: API, coding agent and web. The figures come from the abstract. The source does not name model versions, give a per-model breakdown, or say how many papers came from each venue.
Risks and caveats
The source does not say whether the appendix criticisms were valid or wrong, only that they concern fine-grained technical details not central to the paper's contributions. The 50% figure is a share of the weaknesses the models list, not a share of the paper text. 'About 50% or more' is the only LLM figure given, and no fix for reviewing policy is proposed.