Glonce

Continual Search improves GPT-5.5's failure-diagnosis F1 score by more than 40%

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact65Novelty75Relevance95Surprise50Players45Credibility85
Continual Search improves GPT-5.5's failure-diagnosis F1 score by more than 40%

AI agents that carry out long, multi-step tasks generate execution logs too large for a person to review by hand, so diagnosing why a given run failed, known as root-cause attribution (RCA), has to be automated to be usable. Existing automated RCA methods that lean on large language models still have low diagnostic accuracy, and the problem gets worse as execution traces grow longer. The evidence that actually explains a failure is often sparse, scattered across actions far from the failure, and disconnected from the point where the failure becomes visible, which in practice turns root-cause attribution into a search problem rather than a single judgment call.

Current RCA methods typically hand an LLM judge the whole trace and ask for one diagnosis in a single pass. That one-shot approach works reasonably well on short trajectories, but on longer ones the judge tends to settle on a plausible-looking answer early and never examines the evidence still sitting further along the trace. The paper introduces Continual Search, an iterative framework built to fix exactly that: instead of one pass, it nudges the same judge over successive turns to keep searching until the unresolved diagnostic evidence has been accounted for.

The authors test Continual Search across four existing RCA benchmarks, which the text does not name individually. Because none of those benchmarks contain the kind of massive execution traces that real long-horizon agents actually produce, they also build a new one: MegaRCA-Mix, a set of 50 human-annotated failure trials drawn from long-horizon, execution-heavy tasks.

Across the benchmark suites and multiple model families, Continual Search consistently improves attribution performance over the one-shot baseline. On MegaRCA-Mix specifically, it improves GPT-5.5's F1 score by more than 40%, from 0.349 up to 0.498. The authors report a further pattern on top of that: within the same model family, a lower-tier model using Continual Search can outperform its higher-tier counterpart, which they take as evidence that how well a model searches matters more than its raw scale. The text gives no specific model names or scores behind that particular claim beyond GPT-5.5's own before-and-after F1 numbers, and it names no authors, institution, or publication date.

Key facts

  • Continual Search is an iterative framework that repeatedly prompts an LLM judge, over successive turns, to keep searching for unresolved diagnostic evidence in AI-agent failure logs instead of settling on a one-shot diagnosis.
  • On the researchers' new MegaRCA-Mix benchmark (50 human-annotated failure trials), Continual Search raises GPT-5.5's F1 score for root-cause attribution from 0.349 to 0.498, a gain of more than 40%.
  • The method is evaluated across four existing RCA benchmarks in addition to MegaRCA-Mix, and consistently improves attribution performance across multiple model families.
  • The authors report that, within the same model family, lower-tier models can surpass higher-tier counterparts under Continual Search, arguing effective search matters more than raw model scale, without naming which specific models produced that result.
  • MegaRCA-Mix was built because existing RCA benchmarks lack massive execution traces; it targets long-horizon, execution-heavy agent tasks.

Why it matters

Long-horizon agents already produce more execution log data than any team can review by hand, and that gap grows as agents take on longer, more complex tasks. Root-cause attribution turns a bare pass or fail signal into something a team can act on, but the paper argues automated RCA has been treated as a single judgment call when the evidence that explains a failure is often sparse and scattered far from where the failure becomes visible. Reframing RCA as iterative search rather than a one-shot verdict is what the paper credits for the reported gains: on its own MegaRCA-Mix benchmark, Continual Search improves GPT-5.5's F1 score by more than 40%, from 0.349 to 0.498.

Who it affects

Teams building or operating AI agents for long, multi-step tasks, coding agents, research agents, workflow automation, who need to diagnose failures at a scale no human reviewer can keep up with. It also matters to researchers working on RCA evaluation itself: the paper contributes a new benchmark, MegaRCA-Mix, built specifically because the four existing RCA benchmarks it tests against lack massive execution traces. The text does not name those four benchmarks, so which existing evaluation efforts are directly affected is not specified.

How to use it

This is a research paper describing a method, not a released product. Continual Search is a prompting strategy layered on top of an existing LLM judge: instead of asking for one verdict on the whole trace, it repeatedly prompts the judge, over successive turns, to keep searching for evidence it has not yet accounted for. It is evaluated across four existing RCA benchmarks plus the authors' own MegaRCA-Mix, 50 human-annotated failure trials. The text does not mention any code, dataset, or model release, so there is nothing here yet to install or run.

How solid is it

The headline figure, more than a 40% improvement in GPT-5.5's F1 score on MegaRCA-Mix, from 0.349 to 0.498, is a before-and-after comparison the authors ran themselves on a benchmark they also built themselves. They report the improvement holds across four existing RCA benchmarks and multiple model families, though the text does not name those benchmarks or the other model families involved. The further claim that lower-tier models can surpass higher-tier ones under Continual Search is stated only as a general finding: the text gives no specific model names or scores behind it, beyond GPT-5.5's own before-and-after numbers. The abstract also gives no author names, institutional affiliation, or publication date, so the work cannot be independently placed here.

Risks and caveats

Every result here is self-reported by the paper's own authors, on a benchmark the same authors built, with no independent replication available from this text. The mechanism is described only at a high level, an iterative process that nudges the judge to keep searching over successive turns, without further detail on how that nudging works. No code, data, or model is mentioned as released, so outside teams cannot yet reproduce or apply Continual Search directly from what is given here.