Glonce

Study finds rhetoric alone can sway AI peer reviewers on 4,200 ICLR papers

researchThe Glonce editorsHugging Face🔥70Glonce rating70Impact60Novelty75Relevance88Surprise57Players38Credibility82
Study finds rhetoric alone can sway AI peer reviewers on 4,200 ICLR papers

A research team built a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions to test whether large language models can be swayed by how a paper is written rather than what it reports. Two LLM rewriters altered six rhetorical dimensions of each paper in opposing directions (making the same underlying content read as more or less confident, novel, or well scoped, for example) while preserving the reported scientific content. Five LLM reviewers then scored the resulting manuscripts under both a standard review protocol and a stricter one. The team also tested joint rewriting (multiple dimensions changed at once), recursive rewriting (repeated passes), and reviewer-guided rewriting (where the rewriter used reviewer feedback). The results show that rhetorical sensitivity in AI review is structured, not uniform across the six dimensions. Evidence framing and novelty stance produced the largest positive-to-negative swings in overall assessment (OA) scores, with scope framing forming a weaker second tier; the remaining dimensions had smaller or less stable effects. This pattern held across papers of different human-assessed quality levels. However, the direction and size of score movement depended heavily on the AI reviewer's original score: papers that started with lower scores tended to rise under favorable rhetoric, papers that started with higher scores tended to fall under unfavorable rhetoric, and the clearest contrasts appeared in the middle of the score range. Adding complexity to the rewriting process did not reliably produce bigger effects. Joint rewriting's impact depended strongly on which rewriter model was used, reviewer-guided rewriting did not consistently beat an unguided second rewriting pass, and repeated rewriting produced diminishing returns that varied by configuration. Across all conditions, the researchers found that the rewriter model mainly determined how far apart the two opposing rewritten versions ended up, while the reviewer model determined the size and direction of the resulting score changes. Switching from standard to strict review lowered the mean OA score by 1.36 points, but this did not consistently change how sensitive reviewers were to rhetoric. The authors frame the effect as a form of reward hacking: a way rhetorical presentation can distort scientific evaluation performed by AI reviewers even when the underlying scientific content is unchanged, and argue that evaluation systems need to be made robust to this kind of content-preserving rewriting.

Key facts

  • The corpus contains 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions.
  • Two LLM rewriters altered six rhetorical dimensions in opposing directions while five LLM reviewers scored the results under standard and strict protocols.
  • Evidence framing and novelty stance caused the largest score swings, with scope framing a weaker second tier and the rest smaller or less stable.
  • Score movement depended on the reviewer's original score: low scores tended to rise, high scores tended to fall, with the clearest contrasts in the middle range.
  • More elaborate rewriting workflows (joint, recursive, reviewer-guided) did not reliably produce larger score gains, and strict review lowered mean OA by 1.36 points without changing rhetorical sensitivity consistently.

Why it matters

As AI systems take on a growing role in scientific evaluation, this study documents a concrete failure mode: rhetorical presentation, not just underlying research quality, can shift how an AI reviewer scores a paper. Because the corpus held reported scientific content constant while varying only rhetoric, the score changes measured here are attributable to framing rather than to differences in the actual research.

Who it affects

The findings bear on conference organizers and publishers considering AI-assisted peer review, on researchers who might learn to write in ways that game such systems, and on anyone designing automated evaluation pipelines where language models act as judges of scientific or technical content.

How to use it

The paper is a research study rather than a released tool; there is no product, price, or license attached to it. Its practical use is as evidence for designing AI review protocols: the results suggest reviewers should be tested for sensitivity to specific rhetorical dimensions, especially evidence framing and novelty stance, before being deployed at scale.

How solid is it

The study used a controlled experimental design: the same 120 underlying submissions were rewritten in opposing rhetorical directions by two different LLM rewriters and scored by five different LLM reviewers across standard and strict protocols, with joint, recursive, and reviewer-guided rewriting tested as additional conditions. This built-in comparison across multiple rewriters and reviewers, rather than a single model pair, supports the claim that the observed pattern is structured rather than a quirk of one system. The source text does not name the specific rewriter or reviewer models used.

Risks and caveats

The corpus was derived from a relatively small base of 120 original submissions, expanded to 4,200 through rewriting rather than through 4,200 independently drawn papers. The study also found that results were rewriter-dependent for joint rewriting and configuration-dependent for repeated rewriting, meaning the size of the reward-hacking effect is not fixed but varies with how the rewriting is done and which models are involved.