FARS outperforms rival AI Scientist systems in first LLM peer-review benchmark
A paper proposes and runs a benchmarking protocol that uses large language models as automated peer reviewers to grade AI-generated research papers on four dimensions: originality, scientific rigor, clarity, and significance. The authors took 15 research proposals originally published by FARS, a commercial autonomous AI scientist company, and ran each proposal through four competing AI Scientist frameworks: Sakana AI (versions 1 and 2), CycleResearcher, and Data-to-Paper. That produced 60 generated papers, which the authors then scored alongside FARS's own 15 benchmark papers written from the same proposals, using three independent LLM reviewers: GPT-5.4, Gemini, and Claude.
FARS's own benchmark papers scored a mean of 2.14 to 2.47 on a 1 to 5 scale, against 1.00 to 1.87 for the four competing frameworks. On the Gemini and Claude evaluations specifically, FARS scored more than 2 times higher than the next-best competing system. The authors also checked whether the three LLM reviewers agreed with each other: Gemini and Claude showed strong agreement (correlation rho = 0.907, p < 0.001), and both correlated extremely strongly with a combined synthesis score (rho = 0.961, p < 0.001). GPT-5.4 diverged from the other two, showing much weaker agreement (rho of about 0.32), which the authors read as GPT-5.4 applying different evaluation criteria rather than simply being wrong.
The authors present this as the first quantitative benchmark for AI Scientist systems, and argue that multi-model LLM evaluation offers a scalable, consistent way to assess the quality of autonomous research output at scale.
Key facts
- Four AI Scientist frameworks (Sakana AI v1 and v2, CycleResearcher, Data-to-Paper) each generated papers from the same 15 research proposals, producing 60 papers total, scored alongside 15 benchmark papers from FARS on the same proposals.
- Three independent LLM reviewers (GPT-5.4, Gemini, Claude) scored every paper on originality, scientific rigor, clarity, and significance using a 1 to 5 scale.
- FARS benchmark papers scored a mean of 2.14 to 2.47, versus 1.00 to 1.87 for the four competing frameworks.
- On the Gemini and Claude evaluations specifically, FARS scored more than 2 times higher than the next-best competing system.
- Gemini and Claude agreed strongly with each other (rho = 0.907) and with the synthesis score (rho = 0.961), while GPT-5.4 showed much weaker agreement (rho of about 0.32).
Why it matters
Evaluating AI-generated research papers has been an open problem: there was no established way to compare the output of different autonomous research systems against each other or against human-designed benchmarks. This paper proposes and tests one answer, an automated multi-model LLM peer-review protocol, and the authors describe it as the first quantitative benchmark of its kind for AI Scientist systems.
Who it affects
The comparison directly covers five systems: FARS, whose own papers served as the benchmark standard, and four competing AI Scientist frameworks, Sakana AI (v1 and v2), CycleResearcher, and Data-to-Paper. More broadly, the protocol is aimed at anyone building or evaluating autonomous research-generation systems, since it offers a repeatable way to score paper quality without relying solely on slower human peer review.
How to use it
The source does not describe pricing, access, or tooling for the benchmark itself; it is a research methodology rather than a product. Researchers wanting to compare AI Scientist systems could apply the same protocol: run each system on a shared set of proposals, then score the resulting papers with multiple independent LLM reviewers rather than a single model.
How solid is it
The test set is modest but controlled: the same 15 research proposals were run through all four competing frameworks and compared against 15 matching FARS papers, for 75 papers total. Using three separate LLM reviewers, rather than one, lets the authors check consistency directly, and they report that Gemini and Claude agreed strongly with each other and with a combined synthesis score, which supports treating automated LLM review as a reasonably reliable measure here.
Risks and caveats
GPT-5.4 disagreed with the other two reviewers considerably more than they disagreed with each other, which the authors attribute to different evaluation criteria rather than error, but it means the overall result depends partly on which reviewer model is weighted more heavily. FARS also supplied the original research proposals and its own papers serve as the top-scoring comparison point, a setup worth keeping in mind when reading the result. The source gives no per-framework breakdown among the four competing systems, no cost or runtime figures for running them, and does not state whether the review was conducted blind to which system produced which paper.