Glonce

Causal-Discovery-Bench: LLM agents fail at designing experiments, not at inference

researchThe Glonce editorsarXiv🔥60Glonce rating60Impact45Novelty60Relevance100Surprise35Players10Credibility75

As LLM agents are adopted for autonomous research, they have to do more than reason over existing evidence. They must design informative experiments, allocate resources, analyze sequential observations, and decide when to stop and draw conclusions. The paper's authors say these capabilities remain poorly measured, and they introduce Causal-Discovery-Bench to measure them.

The benchmark is a verifiable sandbox built on configurable structural equation models with resource constraints. It evaluates sequential experimental design and causal discovery without domain-semantic cues, so an agent cannot lean on what it already knows about a subject. The code is linked from the abstract at github.com/mshen1019/causal-discovery-bench.

The evaluation covers 3,802 episodes with 13 models from 5 vendors. The authors analyze agent behavior along 40 metrics grouped in seven families.

Agent performance, they report, follows a four-level difficulty hierarchy: single-experiment design; sequential exploration of causal relationships; resource-aware experimentation that balances cost against statistical power; and autonomous resolution of conflicting evidence.

The central finding is a substantial gap between two tasks: causal inference from correctly curated evidence, and navigating causal discovery through autonomous experimentation. Analyzing the agents' action trajectories, the authors attribute 94.5% of their mistakes to experimental design rather than to inference. They say ablation studies further consolidate this. The 94.5% is a share of mistakes, not an accuracy or success rate.

Two interventions were tested. Relaxing resource constraints alone does not reliably resolve failures in sequential discovery. Basic statistical tools, by contrast, substantially improve performance across multiple model families. Advanced tools add little when models lack the prerequisite lower-level capabilities.

Key facts

  • Causal-Discovery-Bench is a verifiable sandbox of configurable structural equation models with resource constraints, designed to test sequential experimental design and causal discovery without domain-semantic cues.
  • The study ran 3,802 episodes with 13 models from 5 vendors and analyzed behavior along 40 metrics in seven families.
  • From action trajectories, the authors attribute 94.5% of agents' mistakes to experimental design rather than to inference, a finding they say ablation studies further support.
  • Performance follows a four-level difficulty hierarchy, from single-experiment design up to autonomous resolution of conflicting evidence.
  • Relaxing resource constraints alone does not reliably fix failures, while basic statistical tools substantially improve performance across multiple model families.

Why it matters

Agents are being adopted for autonomous research, and the authors argue the skills that work needs are poorly measured: designing informative experiments, allocating resources, analyzing sequential observations and knowing when to stop. This benchmark targets those skills directly. Its headline result is that when agents fail at causal discovery, the authors attribute almost all of the mistakes (94.5%) to how they design experiments, not to how they reason over the results. That separates two abilities that are easy to conflate when a model is only given curated evidence to analyze.

Who it affects

Researchers and teams building or evaluating LLM agents for autonomous research and scientific discovery are the most direct audience. The evaluation covers 13 models from 5 vendors, though the abstract does not say which models were tested.

How to use it

The benchmark code is linked from the abstract at https://github.com/mshen1019/causal-discovery-bench. The abstract's practical pointers are these: giving agents basic statistical tools substantially improves performance across multiple model families, whereas relaxing resource constraints alone does not reliably fix failures in sequential discovery. Advanced tools help little if a model lacks the lower-level capabilities they build on.

How solid is it

The setup is sizeable: 3,802 episodes, 13 models, 5 vendors, 40 metrics in seven families. The 94.5% figure comes from analysis of action trajectories, and the authors say ablation studies further consolidate it. The abstract names no authors or institutions, does not name the 13 models or 5 vendors, and gives no per-model results or rankings. It gives no absolute accuracy or success rates, and no size for the improvement from basic statistical tools. It does not say whether the paper is peer-reviewed.

Risks and caveats

The 94.5% is a share of mistakes, not a success rate, so it says where agents go wrong rather than how often they succeed. The tasks are synthetic structural equation models run without domain-semantic cues, so results may not carry over directly to real research settings; the abstract does not address that. The abstract does not say which basic or advanced tools were tested, so the tool findings are best read as directional. It also does not say which models perform best or worst.

“Relaxing resource constraints alone does not reliably resolve failures in sequential discovery.”

— Causal-Discovery-Bench paper abstract (arXiv 2611.09934)