Glonce

Claude Code, Codex and Cursor pick the same tool in just 42% of cases

researchThe Glonce editorsHacker News🔥80Glonce rating80Impact62Novelty75Relevance100Surprise70Players75Credibility80
Claude Code, Codex and Cursor pick the same tool in just 42% of cases

A benchmark published on the armature.tech blog set three coding agents, Claude Code, Codex and Cursor, loose on real engineering tasks inside synthetic codebases, then tracked exactly which third-party tools and services each agent reached for. The team first analyzed thousands of public GitHub repositories for their mix of languages, frameworks, third-party services, deployment platforms, team sizes and codebase age, then adjusted that sample so open-source-heavy startups did not crowd out the patterns seen at larger companies. Coding agents were then used to build 75 repositories matching that corrected profile, spanning 10 programming languages, each with a fake company name, a fake git history and fake API keys, but real lockfiles checked against package registries such as npm.

Each repository received tasks phrased in one of four styles: a vibe-coder, who describes only symptoms and the desired end state and rarely names the tool category; a junior engineer, who usually also names the category; a senior engineer, who is precise about requirements and about what to avoid; and an enterprise engineer, who spells out specific constraints such as compliance and procurement. That produced 1,163 prompt variations, one example being a request to have every generated invoice emailed to the user with a nice message, leaving the agent to find and implement the best solution itself. In 20 to 25% of cases the prompts also added specific details, such as cost or usage volume, to see whether that shifted the outcome. Each run happened in its own disposable sandbox, rotated across three providers (E2B, Blaxel and Daytona) to rule out sandbox-specific effects, and a 'simulated human' played by Gemini 3.7 Flash sat in the loop: it would either accept the agent's top suggestion or ask the agent to pick and implement the best option itself. The authors say that letting an agent implement without ever pausing to ask a question biased it toward building everything in-house, since it had no chance to ask permission to bring in a third-party service; adding the simulated human back in reduced how often the biggest, cloud-native names dominated. In the object-storage tests, for instance, Cloudflare R2 only started beating Amazon S3 once sessions that had previously always defaulted to S3 got that human check. A second, separate instance of Gemini 3.7 Flash then acted as judge, ruling out sessions that were biased by a repository that had already 'pre-chosen' a provider or that never actually settled on one solution (rejecting, for instance, a repository whose only 'solution' was OpenTelemetry on its own, without pairing it to an observability platform), and identifying which vendor each agent ultimately picked from the conversation and the code diff.

Out of 16,893 total runs, the team kept 5,292 sessions across 51 codebases and 18 sectors as valid, and is publishing that first wave now along with the full traces behind it; the remaining 10,000-plus runs were not discarded and may appear in a later wave. The clearest result is that the three agents disagree far more than they agree: Claude Code, Codex and Cursor pick the identical tool in only 42% of matched cases, and in the voice-agents category, for example, Claude Code goes with Twilio, Codex with the OpenAI Realtime API and Cursor with Vapi. Part of the divergence traces to how each agent gathers information. Cursor bases its decision on the web in two-thirds of sessions. Codex searches the web in 94% of sessions and, nine times out of ten, narrows that search with operators such as site: to focus on a specific trusted domain. Claude Code instead leans on its own priors and only searches the web in about 30% of cases, though when it does, it reads roughly three times as many pages as Codex does; in newer categories such as sandboxes, where its priors are weaker, that web-search rate climbs to about 80%. Claude Code also defaults to building a custom, in-house solution far more often than the other two: about 19% of its sessions, against 10% for both Codex and Cursor.

Which vendor wins can hinge entirely on the repository rather than on the task itself. Given the identical invoice-emailing request, four different repositories in four different languages produced four different winning email providers: Resend on TypeScript (55 of 89 runs), SendGrid on Python (22 of 24), Postmark on Go (20 of 24) and Azure Communication Services on Java (22 of 23). Vercel won on TypeScript repositories, including every session where the repository used Next.js, but was never recommended at all on Python repositories, where Render dominated instead.

Being mentioned constantly is no guarantee of being picked. PayPal comes up in conversation 139 times across the payment-provider sessions but is never chosen; Stripe wins 124 of those same 139 sessions. Adyen fares similarly, mentioned 175 times and picked only 3 times. LangChain is the single most-cited framework in the whole dataset, named 194 times, yet it is picked in just 4 sessions. Netlify is mentioned 152 times as a deployment option and picked 6 times, and Supabase is the most-mentioned database candidate at 242 mentions while still being, in the authors' words, largely dominated by Neon.

Small details on a vendor's own pages can flip the outcome. Agents regularly passed over Mailgun in favor of Postmark specifically after reading that Mailgun's free plan offers only '1-day retention.' Supabase, according to the authors, tends to lose specifically when agents are hunting for a plain database and instead find its bundled pricing pushing extra BaaS features (authentication, storage, realtime) they did not ask for. Looking across the 5,292 valid sessions, 388 mentioned platform-management overhead and 195 mentioned cost; the authors say that in many of those cases, the deciding factor looked more like how the vendor had presented the information than an actual disqualifying fact about the product.

Some categories are lopsided and some are close contests. Stripe wins as payment provider in 9 cases out of 10, losing only in EU-regulated scenarios where more specialized players, Paddle and Mollie, are chosen instead. Neon wins as database in 66% of sessions, ahead of cloud-native options from Azure and AWS. For file storage, Amazon S3 dominates at 45%, with Azure and GCP tied at 20% each. Email is a closer race: Resend and Postmark post install rates of 35.6% and 27.4% respectively across the wider dataset.

The authors frame this as the first of several planned releases: they say they have only pulled a fraction of the learnings still sitting in the traces, plan to run further experiments, and are inviting readers to send in the questions they would like tested next, at contact@armature.tech. All the traces behind the published sessions, along with a leaderboard and the underlying analyses, are now public.

Key facts

  • The team ran 16,893 sandboxed sessions across 75 synthetic repositories in 10 languages, and is now publishing the 5,292 of those sessions (across 51 codebases and 18 sectors) that its Gemini 3.7 Flash judge rated valid.
  • Claude Code, Codex and Cursor pick the same tool in only 42% of matched cases; Claude Code searches the web in about 30% of sessions (versus Codex's 94%) but reads roughly three times as many pages when it does, rising to about 80% of sessions in newer categories such as sandboxes.
  • Claude Code defaults to building its own in-house solution far more often than the other two agents: about 19% of cases, versus 10% for Codex and Cursor.
  • Being mentioned is not the same as being picked: PayPal comes up 139 times but is never chosen (Stripe wins 124 of those same sessions), and LangChain is the most-cited framework at 194 mentions yet is picked only 4 times.
  • The winning vendor for an identical task can flip with the repository's language alone: for the same invoice-emailing prompt, Resend wins on TypeScript (55 of 89 runs), SendGrid on Python (22 of 24), Postmark on Go (20 of 24) and Azure Communication Services on Java (22 of 23).

Why it matters

The study's central finding is that three agents given the identical task do not converge on the identical answer: they pick the same tool in only 42% of matched cases, and which vendor wins can flip just because the repository is written in a different language. That matters for anyone letting an agent choose infrastructure or services unsupervised, since the result is not fully predictable from the task alone. The study also shows that being well known is not the same as being chosen: PayPal, Adyen, LangChain, Netlify and Supabase are among the most frequently mentioned names in their categories and are still rarely picked, which cuts against the assumption that vendor familiarity drives agent recommendations. And the methodology itself carries a finding: adding a 'simulated human' checkpoint measurably changed outcomes, with Cloudflare R2 only starting to beat Amazon S3 in object storage once that human check was present, meaning how an agent is supervised shifts vendor outcomes as much as which agent is used.

Who it affects

Developers and teams that let Claude Code, Codex or Cursor pick infrastructure or third-party services for real projects. The vendors named as frequent winners or frequent also-rans, including Stripe, PayPal, Adyen, Paddle, Mollie, LangChain, Netlify, Vercel, Render, Supabase, Neon, Azure, AWS, GCP, Amazon S3, Cloudflare R2, Resend, SendGrid, Postmark, Azure Communication Services, Mailgun, Twilio, the OpenAI Realtime API and Vapi. And researchers or vendor teams trying to understand how LLM-based coding agents make adoption decisions on their own.

How to use it

The traces behind the 5,292 published sessions are public now, together with a leaderboard laying out the results, analyses and key learnings by sector. A team curious about its own category, email providers or object storage, for instance, can look up how each agent reasoned before settling on a vendor and weigh that against its own requirements rather than accepting a single agent's suggestion outright. The authors say the roughly 10,000 runs not included in this first wave were not discarded and may surface later, and they are asking readers to send the questions they would like tested next to contact@armature.tech.

How solid is it

The design has real rigor: 75 repositories across 10 languages, built to match a distribution deliberately corrected so open-source-heavy startups did not skew the panel; fake company details but real, registry-checked lockfiles; four distinct persona-based prompt styles yielding 1,163 variations; runs rotated across three separate sandbox providers specifically to rule out a sandbox-specific effect; and a judge model applying stated validity criteria before any session counted, which is why only 5,292 of 16,893 raw runs made this first release. Set against that, the piece never names the organization or any author behind the work, the only self-identification being a contact address at armature.tech; no date is given for when the experiments ran; the models actually powering Claude Code, Codex and Cursor inside the tests are never specified, only the orchestrator and judge model (Gemini 3.7 Flash); the full rosters behind the '18 sectors' and '51 codebases' are not listed, just a handful of named examples; and no formula is given for how 1,163 prompt variations produced 16,893 total runs.

Risks and caveats

None of the reported figures, including the 42% cross-agent agreement rate, come with a margin of error or confidence interval, so they read as point estimates from a first wave rather than statistically bounded results. Explanations for why a specific vendor lost, such as agents reacting to Mailgun's '1-day retention' wording or to Supabase's bundled features, are the authors' own reading of the transcripts rather than a controlled test isolating that one variable. The interactive leaderboard the piece points to for the full results and traces is not part of the captured text and so could not be checked here. And the term 'cells,' used for the 42% agreement figure, is never defined in the piece, so it is not clear whether that denominator counts sectors, repositories, language variants, or some combination of the three.

“Now I need that each invoice that we generate gets sent to the user's email address with a nice message, find the best solution and implement it.”

— example task prompt used in the benchmark