Glonce

Ai2 study finds LLM novelty judges swing wildly on small prompt changes

researchThe Glonce editorsHugging Face🔥61Glonce rating61Impact45Novelty60Relevance95Surprise40Players40Credibility75
Ai2 study finds LLM novelty judges swing wildly on small prompt changes

Ai2 has released Old Ideas, Novel Problems, a study of how reliably large language models can judge whether a scientific research idea is novel. With the paper come a benchmark, data and code for testing novelty judges, plus an automated pipeline that researchers can use to build new evaluation sets.

The motivation is Ai2's own work on AI systems for scientific discovery (DataVoyager, AutoDiscovery, Theorizer and Asta). Generating ideas is only part of the job; the ideas also have to be genuinely new, and that call is often handed to another LLM, which Ai2 calls a novelty judge. There is no standard way to build one: teams pick a model, prompt, evaluation format, reasoning budget and sometimes a literature-retrieval system. Ai2 notes that such judges are often validated only on human-written papers, not on the AI-generated ideas they are meant to evaluate.

To build the benchmark, Ai2 mined ICLR 2026 peer reviews for passages where reviewers explicitly discussed a contribution's originality, rather than treating acceptance as a proxy for novelty. The high-novelty pool has 154 human-authored ideas from accepted papers in the top-rated 10% of their ICLR research area, where reviewers positively identified novelty and none disputed it. The lower-novelty pool has 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none positively identified novelty. In a check of the review-mining process, an annotator agreed with the extracted polarity in 47 of 50 cases. These two pools form the Human-Only setting. A second setting, Human+Generated, pairs the same 154 human ideas with 154 ideas from a simple LLM generator that got only a research area and an instruction to propose a novel idea, with no literature search, tools, iteration or feedback. The generated ideas are assumed to be less novel; Ai2 says these are weak labels, not verified for every idea. An expert reviewed 30 Human+Generated pairs, weighted toward cases the strongest judge got wrong, preferred the human idea in all 30 and judged none of the generated ideas novel. Ai2 also rewrote human and AI ideas into a common structure to test whether judges were just reacting to writing style.

Six judges were tested: GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6. They worked pointwise (absolute novelty ratings) and pairwise (pick the more novel of two). Ai2 varied one factor at a time: prompt, reasoning budget, access to related papers, idea format, how many judgments were combined, and, for AI ideas, which model generated them. Prompting caused the largest shifts. In one experiment, changing just a few words moved a model's accuracy at picking the more novel idea by more than 50 percentage points. With GPT-5.4 on Human+Generated pairs, a prompt telling the model that reviewers found exactly one idea in each pair novel gave 93.5% accuracy; the simpler prompt asking which idea was novel dropped it to 40.9%, below chance, and the model changed its verdict on more than half of the pairs. The same prompt change moved accuracy by only about seven percentage points on Human-Only, though Claude Opus 4.6 still shifted by 21 points there. Ai2 reads this as a sign that a judge can look stable on human-authored work but behave very differently on AI-generated ideas.

There was no universally better prompt either. When judging ideas individually, dropping an instruction that warned against treating trivial or incremental contributions as novel helped some judges and hurt others. When comparing ideas from two AI generators (Claude Opus 4.5 against Claude Sonnet 4.5, and GPT-5.4 against GPT-5.1), roughly a quarter to a third of verdicts changed on average across the six judges under the tested prompt changes. That disagreement can be measured without assuming which generator is better.

More search and reasoning did not reliably help. An agentic judge, GPT-5.6 Sol with an extra-high reasoning budget and live iterative search restricted to arXiv, scored 0.78 macro-F1 on Human+Generated. That is 11 points behind Claude Opus 4.6 at 0.89, which saw five pre-retrieved abstracts, and in about the same range as simpler Claude Opus 4.5 and GPT-5.4 setups, at roughly 10x the cost per decision at current API prices. All judges in that comparison ran once per idea.

Purpose-built evaluators did no better. A simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls reached 0.854 macro-F1 at a total evaluation cost of $2.59. Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66. Sakana AI's AI Scientist, whose novelty evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both used Claude Opus 4.6 with one call per decision. The simple baseline was more accurate and considerably cheaper than either.

Ai2's conclusion is that LLM-based novelty evaluation remains a major open problem. It suggests future work should test judges on the kinds of ideas they will actually evaluate, measure sensitivity across prompts and models, improve how prior literature is used, and build benchmarks that evolve as the literature grows.

Key facts

  • Across six LLM judges (GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, Claude Opus 4.6), small evaluation changes produced dramatically different novelty verdicts; in one experiment a few reworded prompt words shifted accuracy by more than 50 percentage points.
  • GPT-5.4's accuracy on Human+Generated pairs fell from 93.5% to 40.9% (below chance) when the prompt changed; the same change moved Human-Only accuracy by only about seven points.
  • An agentic GPT-5.6 Sol judge with live arXiv search scored 0.78 macro-F1 against 0.89 for Claude Opus 4.6 with five pre-retrieved abstracts, at roughly 10x the cost per decision.
  • A simple Claude Opus 4.6 prompt (low reasoning, majority vote of three) scored 0.854 macro-F1 for $2.59; Idea Novelty Checker scored 0.642 at $30.66 and Sakana AI's AI Scientist 0.772 at $82.05.
  • The benchmark is built from ICLR 2026 reviews (154 high-novelty and 145 lower-novelty human ideas); Ai2 is releasing the benchmark, data, code and a pipeline for building new evaluation sets.

Why it matters

Automated scientific ideation only works if something can tell a genuinely new idea from a repackaged one, and that job is increasingly given to another LLM. Ai2's results say that judge is fragile exactly where it is being used: on AI-generated ideas. Judges that looked fairly stable on human-written papers flipped verdicts on more than half of GPT-5.4's pairs after a prompt tweak. The study also undercuts two common assumptions, that adding retrieval or more reasoning makes a judge better, and that a purpose-built evaluator beats a plain prompt.

Who it affects

Teams building AI research-idea generators and the evaluators that rank them, including comparisons between ideation systems, where roughly a quarter to a third of verdicts changed on average with the judge prompt. It also touches researchers who rely on LLM novelty scores, and developers of dedicated tools such as Idea Novelty Checker and Sakana AI's AI Scientist, both of which scored below a simple prompted Claude Opus 4.6 baseline on this benchmark.

How to use it

Ai2 is publishing the benchmark, data and code for testing novelty judges, plus an automated pipeline for building new evaluation sets from peer reviews; the post suggests applying it to new reviews, other scientific communities and future judges. For anyone running a novelty judge, the practical lesson is to test it on the kind of ideas it will actually score and to measure its sensitivity across prompts and models, as Ai2 recommends. On cost, the cheapest strong setup in the study was a Claude Opus 4.6 prompt with low reasoning and a majority vote over three calls, at $2.59 total for the Human+Generated evaluation.

How solid is it

This is first-party research from Ai2, published as a blog post alongside a paper. The benchmark labels come from mined ICLR 2026 reviews, and an annotator agreed with the extracted novelty polarity in 47 of 50 checked cases. An expert reviewed 30 Human+Generated pairs, preferred the human idea in all 30 and judged none of the generated ideas novel. Ai2 also stresses that many core findings do not depend on the labels: if two reasonable setups give opposite answers on the same pair, that disagreement itself shows instability.

Risks and caveats

The labels in the Human+Generated setting are weak: Ai2 assumed the generated ideas were less novel and says they are not verified for every idea. The generator was deliberately basic, with only a research area and a request for a novel idea, so it is a conservative comparison. The expert review covered only 30 pairs, heavily weighted toward cases the strongest judge got wrong. The 93.5% to 40.9% drop is a single GPT-5.4 experiment, and the "more than 50 points" shift comes from one experiment as well. The agentic and purpose-built comparisons ran one call per decision on one dataset, and the cost ratios reflect current API prices. Ai2 itself frames the work as a starting point, with much of the problem unresolved.

“Our results suggest that LLM-based novelty evaluation remains a major open problem.”

— Ai2, Old Ideas, Novel Problems blog post