Glonce

Study of 127,833 clinician queries finds LLM benchmarks miss real use

researchThe Glonce editorsHugging Face🔥62Glonce rating62Impact55Novelty65Relevance90Surprise40Players10Credibility75
Study of 127,833 clinician queries finds LLM benchmarks miss real use

LLM assistants are being rolled out to clinicians across health systems, and judgments about their readiness rest largely on benchmark scores, most of them derived from examination questions or curated cases. The authors of this paper start from a simple point: a benchmark predicts performance in deployment only to the extent that its items resemble real use, and whether benchmarks reflect the work these systems actually receive has rarely been measured.

To measure it, they analysed 127,833 queries sent by 6,342 physicians, advanced practice providers and nurses in 35 specialties to an institutional assistant during an eight-month roll-out. Each query was characterised with RCQ-Map, a clinician-validated framework grounded in taxonomies of clinical questions and of LLM evaluation. It records a query's task, intent, answerability, missing information and potential harm.

Real use looked quite different from what benchmarks emphasise. Documentation and administration made up 36.2% of queries and knowledge retrieval 28.9%, together nearly two-thirds of use. Diagnosis was 3.7%. More than a third of queries could not be answered well as posed.

The authors then applied RCQ-Map to 58 public benchmarks drawn from major evaluation suites and frontier model reports, and assembled them into the Clinical AI Benchmark Atlas. The median benchmark contained no documentation requests and shared 31% of the task mix of real use, which is less than an even spread across task categories would give. Benchmarks in suites designed to resemble clinical practice were individually no closer to real use than those used in frontier model reports.

The authors conclude that benchmark scores say little about how clinical AI performs on most of the work it is actually given, and that evaluation should be matched to real clinical use.

Key facts

  • The study analysed 127,833 queries from 6,342 physicians, advanced practice providers and nurses in 35 specialties, sent to an institutional LLM assistant over an eight-month roll-out.
  • Documentation and administration (36.2%) and knowledge retrieval (28.9%) made up nearly two-thirds of use; diagnosis was 3.7%.
  • More than a third of queries could not be answered well as posed.
  • Across 58 public benchmarks, the median one contained no documentation requests and shared 31% of the task mix of real use, less than an even spread across task categories would.
  • Benchmarks from suites designed to resemble clinical practice were individually no closer to real use than those in frontier model reports.

Why it matters

Health systems judge whether an LLM assistant is ready for clinicians largely by benchmark scores. This paper tests that habit directly: it puts what clinicians actually ask next to what benchmarks actually test. The gap is large. Real use is dominated by documentation, administration and knowledge retrieval, while diagnosis is 3.7% of queries. The authors say benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given.

Who it affects

Health systems and clinicians who are deploying or being offered LLM assistants, and anyone deciding readiness from benchmark results. It also bears on the people who build clinical benchmarks and on model developers whose frontier model reports use them: the authors found those benchmarks no closer to real use than suites built to resemble clinical practice.

How to use it

The practical takeaway is the authors' own: match evaluation to real clinical use. In concrete terms, the study offers RCQ-Map, a way to label each query by task, intent, answerability, missing information and potential harm, and the Clinical AI Benchmark Atlas, which collects 58 public benchmarks characterised the same way. A team could compare the task mix of its own assistant's traffic with the task mix of the benchmark it relies on. The abstract does not say whether the paper is peer-reviewed or whether the data or tool are released.

How solid is it

The evidence is a large empirical analysis: 127,833 real queries from 6,342 clinicians across 35 specialties, and 58 benchmarks classified with a clinician-validated framework. The 31% is a task-mix overlap for the median benchmark, not an accuracy or performance figure. The abstract names no authors, institution, health system or hospital, and does not name the institutional assistant or the underlying model. No benchmark is named and no model scores are reported.

Risks and caveats

The usage figures describe one institutional assistant over an eight-month roll-out, so the exact percentages are specific to that deployment. The abstract gives shares only for documentation and administration, knowledge retrieval and diagnosis, and reports no harm rates, although RCQ-Map records potential harm. It also does not say how many of the 58 benchmarks came from clinical-practice suites versus frontier model reports. Low overlap shows that benchmarks and real use differ; the paper does not report how any model scores on the work clinicians actually send.

“Benchmark scores therefore say little about how clinical AI performs on most of the work it is actually given”

— From the paper's abstract