Glonce

IdeaTaste benchmark finds LLMs trail human researchers on research taste

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact45Novelty60Relevance95Surprise40Players15Credibility75
IdeaTaste benchmark finds LLMs trail human researchers on research taste

The paper starts from a gap it says is poorly understood. Research taste, meaning the judgment of what to read, which problems to pursue and which methods to try, increasingly shapes the work of AI scientists, yet it is far less understood than their ability to execute research.

To measure it, the authors introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. The experiments span six contemporary language models and four research areas, from established to emerging fields, and cover more than 90,000 proposals in total.

The main result is a large shortfall. LLM agents fall well short of human communities both in the literature they cite and in the ideas they state, reaching about 26% of the human level on citation matching and only 18% on idea matching. Both figures are shares of the human level, not absolute accuracies.

The pattern is more specific than a general weakness. LLM populations can cite as diversely as a community and take up its tasks and methods, yet they rarely arrive at the concrete approaches and research plans the community actually pursues. Neither richer field context nor larger populations substantially close the gap.

Blinded expert review points the same way. AI proposals trail the 2026 papers least in presentation and most in expected success, which echoes the gap in concrete approaches. Proposals whose ideas no human paper shares fare worst, which the authors say suggests weaker directions rather than overlooked opportunities.

From this the authors conclude that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. They release IdeaTaste as an open, continuously updated leaderboard at cxcscmu.github.io/IdeaTaste that benchmarks contemporary AI proposals against future publication cycles.

Key facts

  • IdeaTaste scores an LLM by how closely proposals it writes from a literature snapshot frozen at 2025 match what the same communities published in 2026.
  • The experiments cover six language models, four research areas and more than 90,000 proposals in total.
  • LLM agents reach about 26% of the human level on citation matching and only 18% on idea matching.
  • Richer field context and larger populations of models do not substantially close the gap; the authors locate the bottleneck in turning directions into concrete approaches.
  • In blinded expert review, AI proposals trail 2026 papers least in presentation and most in expected success.

Why it matters

AI systems are increasingly asked not just to run experiments but to decide what is worth researching. The authors say that judgment, which they call research taste, is far less understood than the ability to execute research. IdeaTaste gives it a measurable form by checking proposals against what human communities actually published a year later. The headline finding is that models find the right neighbourhood but not the right plan: they cite diversely and pick up a community's tasks and methods, but rarely land on the concrete approaches it pursues.

Who it affects

Teams building AI scientist or research-agent systems are the most direct audience, since the result concerns the quality of the ideas such agents propose. Researchers who study how to evaluate AI-generated research have a new benchmark and a public leaderboard to compare against. The paper also bears on anyone deciding how much to trust LLM-suggested research directions, because the expert review rated the proposals lowest on expected success.

How to use it

IdeaTaste is released as an open, continuously updated leaderboard at https://cxcscmu.github.io/IdeaTaste. It benchmarks contemporary AI proposals against future publication cycles, so a new system can be compared with the human work of a later year once that work appears. The authors' findings also suggest where to aim effort: the gap lies in turning a direction into a concrete approach, not in finding the direction.

How solid is it

The scale is substantial: six language models, four research areas and more than 90,000 proposals. The design checks proposals against real later publications and adds blinded expert review, and both point the same way. The material here is the paper's abstract, so some details cannot be assessed: the six models and four areas are not named, the way citation and idea matching are computed is not described, and the number of expert reviewers is not given. The 26% and 18% figures are shares of the human level, and the abstract does not say whether they are means or maxima across the six models. No per-model results are given.

Risks and caveats

The benchmark treats the 2026 publications of a community as the reference for good taste, so a proposal that departs from what was published is scored as a miss. The authors read the worst scores for proposals with ideas no human paper shares as a sign of weaker directions rather than overlooked opportunities; that is their interpretation of the expert review. The percentages are relative to a human baseline and should not be read as absolute accuracy. The conclusions come from the authors' own benchmark, and the abstract names no independent replication.

“These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue.”

— IdeaTaste paper abstract