Glonce

Claude Opus 4.6 produces 30x more errors than GPT-5.4 in new agent benchmark

researchThe Glonce editorsarXiv🔥75Glonce rating75Impact65Novelty72Relevance95Surprise60Players85Credibility82

A newly published dataset, OpenDiscoveryTrace, targets a gap in how autonomous AI science agents get evaluated. According to its authors, existing benchmarks for such agents score only the final output, generated code, a hypothesis or a finished paper, while discarding the reasoning process that produced it. That omission, they argue, makes it impossible to audit an agent's scientific methodology, diagnose why it failed or tell systematic reasoning apart from a lucky guess.

OpenDiscoveryTrace itself is a public collection of 558 complete AI agent trajectories that record how each model reasoned, not just what it produced. Every trajectory logs a structured trace with nine fields per step, including thoughts, tool calls, observations, errors, revision triggers and self-reported confidence, as an agent works through one of 124 scientific tasks spanning drug discovery, materials science, genomics and scientific literature analysis. The dataset spans seven models: three frontier systems, GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro, each contributing 124 trajectories balanced across domain and difficulty, and four open-weight models, Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini and Qwen2.5-1.5B, each contributing 30. A further 60 trajectories come from live-retrieval variants of the tasks.

A pilot analysis of 363 of these trajectories, scored by an LLM judge, gives the first look at what the traces show. All three frontier models land on comparable success rates, between 84% and 89%, the kind of result that would make them look interchangeable on an output-only benchmark. The trace-level data tells a different story for at least one pair: Claude Opus 4.6 produces 30 times more errors per trajectory than GPT-5.4, a mean of 2.5 against 0.08, a gap the authors report as statistically significant (p < 0.0001, Cliff's delta 0.613). The errors are not the same kind, either. For Claude Opus 4.6, 66.7% of errors are tool misuse; for GPT-5.4, 83.6% of errors are reasoning errors instead.

Alongside the dataset, the authors define five benchmark tasks, with baseline results from logistic regression, random forests, LSTMs and Transformer models, though the abstract does not specify what the five tasks themselves involve. The dataset, its trace schema, the agent harness used to generate it and the benchmark task definitions are all released publicly under a CC BY 4.0 license, meant to support research on process-level evaluation, scientific agent auditing and AI governance.

Key facts

  • OpenDiscoveryTrace publishes 558 complete AI agent trajectories across 124 scientific tasks in drug discovery, materials science, genomics and scientific literature analysis.
  • The dataset covers seven models: three frontier models, GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro, with 124 trajectories each, and four open-weight models, Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini and Qwen2.5-1.5B, with 30 each, plus 60 live-retrieval variants.
  • In a pilot analysis of 363 LLM-judged trajectories, all three frontier models reached comparable success rates of 84% to 89%.
  • Claude Opus 4.6 produced 30 times more errors per trajectory than GPT-5.4, a mean of 2.5 versus 0.08 (p < 0.0001, Cliff's delta 0.613).
  • The two models' errors differ in kind: 66.7% of Claude Opus 4.6's errors were tool misuse, versus 83.6% reasoning errors for GPT-5.4.

Why it matters

Most agent benchmarks report only whether the final answer was right, which is why two systems with near-identical success rates can look interchangeable. OpenDiscoveryTrace tests that assumption directly: by preserving the full step-by-step trace, thoughts, tool calls, observations, errors, revision triggers and self-reported confidence, it lets researchers see how an agent reached a correct or incorrect answer, not just whether it arrived there. The pilot results back up the premise. GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro all succeed at a similar 84-89% clip on the same scientific tasks, yet Claude Opus 4.6 racks up 30 times more errors along the way than GPT-5.4 while still reaching a comparable final answer. A success-rate leaderboard alone would never surface that gap.

Who it affects

The direct audience is researchers building or evaluating autonomous AI agents for science: drug discovery, materials science, genomics and scientific literature analysis are the four domains the tasks cover, and GPT-5.4, Claude Opus 4.6, Gemini 3.1 Pro, Qwen2.5-7B, Mistral-7B-v0.3, Phi-3.5-mini and Qwen2.5-1.5B are the seven models already traced. It also matters to teams deciding which model to trust with an autonomous research workflow: a model that reaches the right answer while most of its errors are tool misuse carries a different operational risk than one whose failures are reasoning mistakes. Process-level evaluation researchers, scientific agent auditors and AI governance researchers are all named explicitly as the intended users of the released trace schema and harness.

How to use it

The dataset, its nine-field trace schema, the agent harness that generated the trajectories and the five defined benchmark tasks are all public under a CC BY 4.0 license, free to reuse with attribution. The authors provide baseline results for the five benchmark tasks using logistic regression, random forests, LSTMs and Transformer models, a starting point to beat rather than evaluation infrastructure a new entrant has to build from scratch; the abstract itself does not spell out what the five tasks involve, only the baselines used to score them. Because the harness is released too, a team could in principle run its own model through the same 124 tasks and generate directly comparable trace data instead of relying only on the seven models already covered.

How solid is it

The headline error-rate comparison rests on solid-looking statistics, a 30x gap with p < 0.0001 and a Cliff's delta of 0.613, but it comes from a pilot of 363 trajectories out of 558 in the full dataset, scored by an LLM judge whose identity and reliability are not described. The comparable-success-rate finding covers all three frontier models, GPT-5.4, Claude Opus 4.6 and Gemini 3.1 Pro, each with 124 balanced trajectories, but the 30x error-count gap and the split between tool misuse and reasoning errors are reported only for the Claude Opus 4.6 versus GPT-5.4 pair; no matching error figures are given for Gemini 3.1 Pro or for any of the four open-weight models. There is also an unexplained bookkeeping gap: summing the abstract's own per-group counts, 124 trajectories for each of three frontier models, 30 for each of four open-weight models, plus 60 live-retrieval variants, comes to 552, six short of the stated 558 total.

Risks and caveats

The abstract does not explain why Claude Opus 4.6's errors skew toward tool misuse while GPT-5.4's skew toward reasoning errors, so the 30x gap is a measured difference, not a diagnosed cause. Whether the same kind of gap shows up for Gemini 3.1 Pro, or for any of the four open-weight models, is not addressed: the headline comparison covers only the Claude Opus 4.6 and GPT-5.4 pair. The open-weight models themselves are represented by just 30 trajectories each, a much smaller sample than the 124 collected per frontier model, so conclusions about them individually should be treated cautiously.