Glonce

GxP-Agent reaches 100% structural match on clinical trial benchmark

agentsThe Glonce editorsarXiv🔥77Glonce rating77Impact75Novelty80Relevance85Surprise65Players55Credibility85

Clinical trial programming, the work of turning study protocols into analysis-ready datasets under CDISC standards, is described as a bottleneck in regulatory submissions. The authors report that LLM-based code generation fails catastrophically on this task: across 11 single-shot attempts with five frontier models, none produced a valid subject-level analysis dataset.

The researchers introduce GxP-Agent, a multi-agent system that encodes regulatory process ordering as a directed acyclic graph (DAG). Instead of asking a model to generate the whole dataset in one pass, GxP-Agent decomposes the task into 15 domain-specific nodes, each handled by a worker agent equipped with pharmaverse skill context, validation gates, and conditional retry.

The system is evaluated on CDISC-Bench, a new execution-based benchmark the authors built from the FDA pilot submission CDISCPilot01, covering 254 subjects and 49 ground-truth ADSL variables. GxP-Agent running on Claude Sonnet 4.6 achieved 100% structural match, all 49 of 49 variables and all 254 records correct, across three independent runs. The best retrieval-augmented baseline reached 59.2% on the same benchmark, and every single-agent or flat multi-agent approach scored 0%.

The DAG topology also carried over to a weaker model: GPT-4.1 reached 59.2% mean structural match when run inside the same DAG structure, compared with 0% under every other architecture tested on it.

The approach generalized to a second dataset type, ADAE (adverse events), using a 9-node branching DAG covering 55 variables and 1,191 records. GxP-Agent reached 100% structural match on ADAE on the first attempt.

The authors conclude that encoding domain process knowledge as graph topology, rather than relying on LLM reasoning alone, is a key enabler for reliable, GxP-compliant clinical trial programming.

Key facts

  • Across 11 single-shot attempts with five frontier models, none produced a valid subject-level analysis dataset for clinical trial programming.
  • GxP-Agent splits dataset generation into 15 domain-specific nodes arranged as a DAG, each run by a worker agent with pharmaverse skill context, validation gates, and conditional retry.
  • On CDISC-Bench (built from the FDA pilot submission CDISCPilot01: 254 subjects, 49 ground-truth ADSL variables), GxP-Agent with Claude Sonnet 4.6 achieved 100% structural match across three independent runs, versus 59.2% for the best retrieval-augmented baseline and 0% for all single-agent and flat multi-agent approaches.
  • The same DAG topology raised GPT-4.1 from 0% under every other architecture to 59.2% mean structural match.
  • The approach generalized to ADAE (adverse events) data: a 9-node branching DAG covering 55 variables and 1,191 records reached 100% structural match on the first attempt.

Why it matters

Clinical trial programming, converting study protocols into CDISC-standard, analysis-ready datasets, is described as a bottleneck in regulatory submissions. The authors report that LLM-based code generation fails catastrophically on this task: across 11 single-shot attempts with five frontier models, none produced a valid subject-level analysis dataset. GxP-Agent is offered as a fix for that gap, and the authors argue that encoding domain process knowledge as graph topology, rather than relying on LLM reasoning alone, is a key enabler for reliable, GxP-compliant clinical trial programming.

Who it affects

Teams that produce regulatory submission datasets under CDISC standards, the format expected in FDA and other regulatory filings, are the direct audience. The benchmark used to evaluate GxP-Agent, CDISC-Bench, is itself built from a real FDA pilot submission, CDISCPilot01, which ties the evaluation to an actual regulatory data format rather than a synthetic one.

How to use it

The text gives no pricing, release plan, or access details for GxP-Agent or CDISC-Bench. What is documented is the architecture: the ADSL task is split into 15 domain-specific DAG nodes, and the ADAE (adverse events) task into a 9-node branching DAG, with each node run by a worker agent carrying pharmaverse skill context, validation gates, and conditional retry.

How solid is it

The headline result, 100% structural match with 49 of 49 variables and 254 correct records, held across three independent runs on CDISC-Bench rather than a single pass. The comparison points are stated directly: the best retrieval-augmented baseline reached 59.2%, and every single-agent or flat multi-agent approach scored 0%. The DAG topology also transferred across models, lifting GPT-4.1 to 59.2% mean structural match versus 0% under every other architecture tested on it, and it reached 100% structural match on the separate ADAE benchmark on the first attempt.

Risks and caveats

The source does not name the five frontier models used in the 11-attempt single-shot baseline, does not name the paper's authors or institutions, and gives no date of submission or publication. No cost, compute, or runtime figures are given, and the text does not specify which validation gates or retry conditions triggered during the runs, so how often the system needed to retry, or how costly that made each run, is not stated.