Paper analyzes 143,209 coding-agent trajectories to compare how agents behave

Paper analyzes 143,209 coding-agent trajectories to compare how agents behave

Standard coding-agent benchmarks mostly score binary task success. The authors of this paper say that treats execution as a black box and hides the behaviors that produce the outcomes.

To look inside, they analyze a corpus of 143,209 execution trajectories from 125 agent configurations, which span 117 unique models. Because agents use different interfaces, the authors built a rule-based parser that maps raw execution traces to high-level actions such as repository search, file editing and test execution. They audited the parser's predictions against a six-model consensus reference on 500 events.

Three findings stand out. First, agents with similar solve rates can still differ substantially in how they explore, modify and verify code. Second, aggregate behavioral profiles can retrieve the corresponding agent stack across disjoint repository sets and programming languages, without using task outcomes, token counts or tool names as features. The authors add a caveat: task composition is not fully controlled in the cross-repository setting. Third, analyses of frontier models and agent scaffolds show that scaffold choice and reasoning effort systematically shift agents' search, editing and testing behavior.

The authors release the deterministic parser, the full annotation and analysis pipeline, and an annotated subset of the trajectories.

Key facts

  • The corpus holds 143,209 execution trajectories from 125 agent configurations spanning 117 unique models.
  • A rule-based parser maps raw traces to high-level actions such as repository search, file editing and test execution, and was audited on 500 events against a six-model consensus reference.
  • Agents with similar solve rates can differ substantially in how they explore, modify and verify code.
  • Behavioral profiles retrieve the agent stack across disjoint repository sets and programming languages, though task composition is not fully controlled in the cross-repository setting.
  • Scaffold choice and reasoning effort systematically shift search, editing and testing behavior; the parser, pipeline and an annotated subset are released.

Why it matters

Benchmarks that report only whether a task was solved say little about how an agent got there. This work argues for looking at behavior: two agents with similar solve rates can explore, modify and verify code in quite different ways. A behavioral view gives a way to compare agents beyond a single success number.

Who it affects

Researchers who build or evaluate coding agents are the main audience, along with teams that choose between scaffolds or reasoning-effort settings. The released parser and annotated trajectories are aimed at people who want to study agent behavior themselves.

How to use it

The authors release the deterministic parser together with the full annotation and analysis pipeline and an annotated trajectory subset. Because the parser maps traces from different agent interfaces to a shared set of high-level actions, it can be used to compare agents that expose different tools.

How solid is it

This is a preprint-style abstract with a large corpus: 143,209 trajectories, 125 configurations, 117 models. The parser was checked on 500 events against a six-model consensus reference. The abstract gives no agreement figure for that check, so its accuracy cannot be judged from the text. The retrieval result carries the authors' own caveat about task composition.

Risks and caveats

The cross-repository retrieval result comes with the admission that task composition is not fully controlled, so it may partly reflect which tasks were run. The parser is rule-based and was audited on a sample of 500 events. The abstract does not state the size or direction of the scaffold and reasoning-effort effects.