Glonce

Benchmark Radar catalogs 1,283 benchmark records

researchThe Glonce editorsHugging Face🔥66Glonce rating66Impact60Novelty55Relevance95Surprise35Players65Credibility85
Benchmark Radar catalogs 1,283 benchmark records

A new paper presents Benchmark Radar, a living database and search engine for discovering and retrieving AI benchmarks. The authors frame the problem directly: people building or evaluating large language models and other AI systems need to find relevant evaluations, locate the underlying benchmark datasets and code, and understand the settings behind reported scores.

Benchmark Radar covers LLM evaluation, agentic and tool-use benchmarks, coding, reasoning, safety and domain-specific evaluations. It combines daily discovery of new benchmark papers, repositories, datasets and releases with a searchable catalog, mentions of benchmarks in model cards and technical reports, and score histories. The system retains source identities and citations, so readers can inspect candidate benchmarks and the evaluation evidence behind them.

The paper states its scale plainly. Daily discovery draws on 37 sources: 13 direct connectors and 24 first-party research and engineering feeds. The catalog holds 1,283 source records, drawn from 4 benchmark catalogs, with 12,916 numeric observations recorded across 790 of those records.

Beyond building the system, the paper describes its collection and retrieval methods, audits the full catalog, and examines benchmark saturation, adoption trends and the limits of comparing scores across different benchmarks. It also walks through a worked example of a complete prior-art search, showing how to query the catalog and inspect benchmark evidence when designing a new evaluation.

Alongside the paper, the authors release a web dashboard that includes a benchmark leaderboard, a Pareto frontier view plotting score against measured use, saturation and trend views, daily feeds and downloadable evidence, plus a command-line interface for offline queries and reproducible analysis.

Key facts

  • Benchmark Radar is a living database and search engine for discovering and retrieving AI benchmarks across LLM evaluation, agentic and tool-use, coding, reasoning, safety and domain-specific evaluation.
  • Daily discovery draws on 37 sources, 13 direct connectors and 24 first-party research and engineering feeds, to find new benchmark papers, repositories, datasets and releases.
  • The catalog holds 1,283 source records drawn from 4 benchmark catalogs, with 12,916 numeric observations recorded across 790 of those records.
  • Every catalog entry keeps its source identity and citations, so a benchmark's reported score can be traced back to the evidence behind it.
  • The release includes a web dashboard (leaderboard, a Pareto frontier view of score against measured use, saturation and trend views, daily feeds, downloadable evidence) and a command-line interface for offline queries.

Why it matters

The paper states the need directly: people building or evaluating large language models and other AI systems have to find relevant evaluations, locate the datasets and code behind them, and understand the settings behind a reported score. Benchmark Radar is built around that need. By retaining source identities and citations for every catalog entry, it lets readers inspect a candidate benchmark and the evaluation evidence behind it rather than only seeing a final number.

Who it affects

This is aimed directly at benchmark researchers and at developers of large language models and other AI systems, the two groups the paper names as needing to locate evaluations and understand what sits behind a reported score. It is also aimed at anyone designing a new benchmark: the paper's worked example walks through exactly that case, using the catalog to run a prior-art search and inspect existing evidence before building something new.

How to use it

The release includes a web dashboard: a benchmark leaderboard, a Pareto frontier view that plots score against measured use, saturation and trend views, daily feeds and downloadable evidence. For querying without a browser, there is also a command-line interface for offline queries, and the underlying analysis is reproducible. The text gives no URL or access instructions for either the dashboard or the CLI, so exactly where and how to reach them is not stated.

How solid is it

This is a preprint on Hugging Face's papers feed, not a peer-reviewed publication. What is checkable is the scale it claims: a catalog of 1,283 source records assembled from 4 existing benchmark catalogs, with 12,916 numeric observations logged across 790 of those records, built from a daily discovery process running across 37 sources. The paper also includes its own audit of the full catalog and an examination of benchmark saturation, adoption trends and the limits of score comparisons. At the time of writing, the discussion thread on Hugging Face had 46 points and 3 comments.

Risks and caveats

The text does not compare Benchmark Radar to any other benchmark catalog, leaderboard or tracking system, so how it stacks up against the 4 existing benchmark catalogs it draws from is left unaddressed. It also does not say who builds or maintains the system, and it gives no URL or access instructions for the dashboard or CLI. Benchmark saturation and the limits of comparing scores across different benchmarks, both examined in the paper, apply to the catalog's own contents as well as to any single benchmark inside it.