Glonce

ClinLens benchmark finds AI agents run clinical tasks, answers often wrong

researchThe Glonce editorsarXiv🔥69Glonce rating69Impact60Novelty75Relevance95Surprise45Players25Credibility80

A paper introduces ClinLens, a benchmark of 200 executable tasks built over five linked MIMIC clinical resources: structured electronic health records, clinical notes, electrocardiograms, chest radiographs, and echocardiograms. The tasks are organized in a 4x5 taxonomy that crosses four patient-time scopes with five analysis capabilities, aiming to test whether a coding agent can turn heterogeneous, longitudinal patient data into an auditable, correct analysis rather than just a script that runs.

Each task is built with a program-first reverse-synthesis method: a bounded, semi-raw data package is paired with an evaluator-private reference workflow, and grading checks the required artifacts, the cohort and temporal semantics of the analysis, and the final answer, not just whether the code executes.

On a fixed 126-task suite, the authors evaluated 24 standardized model-scaffold configurations. The single strongest configuration reached 56.3% scope-macro STRICTPASS, meaning it produced the fully correct analysis on just over half the tasks, even though it hit 100% EXECSUCCESS, meaning its code ran without error on every task. For reference, a separately configured coding agent solved 83 of the 126 tasks. Five biomedical systems adapted to run on GPT-4o-mini fared far worse, reaching at most 2.9% scope-macro STRICTPASS. The authors frame the headline result as a substantial gap between submissions that merely run and analyses that are actually correct.

Key facts

  • ClinLens is a benchmark of 200 executable tasks built from five linked MIMIC clinical resources: structured EHR data, clinical notes, ECGs, chest radiographs, and echocardiograms.
  • Tasks follow a 4x5 taxonomy crossing four patient-time scopes with five analysis capabilities, graded by evaluator-private reference workflows that check artifacts, cohort and temporal semantics, and the final answer.
  • On a fixed 126-task suite, the best of 24 standardized model-scaffold configurations reached 56.3% scope-macro STRICTPASS despite hitting 100% EXECSUCCESS.
  • A separately configured coding agent solved 83 of the 126 tasks.
  • Five biomedical systems adapted to GPT-4o-mini reached at most 2.9% scope-macro STRICTPASS.

Why it matters

Clinical data science over longitudinal, multimodal patient records, mixing structured tables, free-text notes, and images like ECGs, chest X-rays, and echocardiograms, is a harder real-world target for coding agents than isolated question answering or generic scripting. ClinLens is designed specifically to separate an agent whose code runs from an agent whose analysis is clinically correct, and its central finding is that these two things diverge sharply in practice.

Who it affects

Researchers building or evaluating clinical and biomedical coding agents, teams designing agent benchmarks more broadly, and anyone treating execution success as a proxy for correctness in scientific or clinical domains.

How to use it

ClinLens is a research benchmark, not a shipped product or service, and the source gives no pricing or licensing terms. It offers 200 tasks across five MIMIC resources with reference workflows kept private from the model under test, so teams building clinical data-science agents can run their own pipeline against it to check whether passing code is also producing the right answer.

How solid is it

The benchmark spans four patient-time scopes and five analysis capabilities in a 4x5 taxonomy, with tasks built via program-first reverse synthesis against evaluator-private reference workflows. The authors tested 24 standardized model-scaffold configurations on a fixed 126-task suite: the strongest reached 56.3% scope-macro STRICTPASS while also achieving 100% EXECSUCCESS. A separately configured coding agent solved 83 of the 126 tasks, and five biomedical systems adapted to GPT-4o-mini reached at most 2.9% scope-macro STRICTPASS.

Risks and caveats

The paper's own headline finding is itself the caveat: 100% EXECSUCCESS alongside just 56.3% scope-macro STRICTPASS on the best configuration shows that code running without error is no guarantee the clinical analysis is correct, a gap that matters directly where a wrong but runnable analysis could mislead. The five biomedical systems adapted to GPT-4o-mini fared far worse, at most 2.9% scope-macro STRICTPASS. The source text gives no author names, institutions, or publication venue.

“These results expose a substantial gap between runnable submissions and correct clinical analyses.”

— the paper