Glonce

RealSWE finds realistic prompts cut coding agent scores by 6.4pp

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact70Novelty78Relevance88Surprise48Players38Credibility82
RealSWE finds realistic prompts cut coding agent scores by 6.4pp

Coding agents are usually graded on SWE-bench style benchmarks, whose tasks are drawn from curated GitHub issues that are long, structured and packed with information. Researchers behind a new paper argue this does not look like how people actually ask agents for help. To measure the gap, they built a six-category information taxonomy and a four-dimension linguistic-style framework, then applied both to real user prompts pulled from SWE-chat and to the problem statements used in SWE-bench Verified and Pro. The comparison is stark: prompts that give only a bare problem statement, with little or no extra context, make up 88% of real user requests but only 7% of benchmark problems. On tone, 87% of real prompts are casually written, against 94% of benchmark problems that are formal. Using these observations, the researchers built RealSWE, a set of 381 multi-variant task families derived from SWE-bench Verified and Pro. Every variant in a family shares the same underlying task and the same gold patch, but the variants differ in what information they include and in how that information is phrased. Running seven contemporary LLMs through RealSWE, the study found that realistic inputs lower average resolution rates by 6.4 percentage points and can change which model ranks highest. A controlled breakdown of the information categories showed that including Desired Behavior and Motivation in a request has a real, significant effect on how well an agent performs, while adding Environment Information or Reproduction Steps mostly just adds tokens without a measurable benefit. Linguistic style, by contrast, produced only small effects that varied by model. The paper's practical takeaway is that stating the desired behavior and the motivation behind a request, something most real prompts leave out, is what substantially improves an LLM's software-engineering performance, more than adding extra procedural detail.

Key facts

  • RealSWE is a set of 381 multi-variant task families built from SWE-bench Verified and Pro, with each variant sharing the same task and gold patch but varying in information content and writing style.
  • Bare problem-statement prompts make up 88% of real user requests (from SWE-chat) but only 7% of SWE-bench Verified/Pro problems; 87% of real prompts are casual versus 94% of benchmark problems that are formal.
  • Testing seven contemporary LLMs, the study found realistic inputs cut average resolution rates by 6.4 percentage points and can change model rankings.
  • Including Desired Behavior and Motivation in a prompt significantly affects agent performance; adding Environment Information or Reproduction Steps mostly adds tokens with no measurable benefit.
  • Linguistic style (formal versus casual phrasing) has only a small, model-dependent effect on results.

Why it matters

Coding-agent leaderboards are built on SWE-bench style benchmarks whose problem statements are curated, detailed and formally written, unlike the sparse, casual requests real users type. This paper quantifies that gap for the first time with a structured taxonomy, and shows the mismatch is not cosmetic: it measurably lowers resolution rates and can reorder which model looks best, which means benchmark rankings built on curated issues may not predict how models perform for actual users.

Who it affects

The findings matter to teams that build and evaluate coding agents, to LLM vendors whose models are ranked on SWE-bench style leaderboards, and to benchmark designers deciding what a representative test task should look like. They also give end users of coding agents a concrete reason to change how they phrase requests.

How to use it

The paper's actionable advice is to write requests that state the desired behavior and the motivation behind the change, since most real prompts omit exactly that information and its absence is what hurts performance most. Padding a request with extra environment details or step-by-step reproduction instructions does not help and mainly adds tokens, so that effort is better spent describing what outcome is wanted and why.

How solid is it

The study builds directly on two established benchmarks, SWE-bench Verified and Pro, generating 381 controlled task families where each variant holds the underlying task and gold patch fixed while only varying information content and style, which isolates the effect being measured. It also draws its picture of real-world requests from actual user prompts (SWE-chat) rather than assumptions, and tests the resulting hypotheses across seven contemporary LLMs. The abstract does not name the authors, their institutions, the seven models evaluated, or give a per-model breakdown of the results, only the 6.4 percentage point average.

Risks and caveats

Only the aggregate 6.4 percentage point drop is reported; there is no per-model breakdown showing which systems are hurt most or how rankings actually shifted. The six information categories and four linguistic-style dimensions are not individually defined beyond the examples named (Desired Behavior, Motivation, Environment Information, Reproduction Steps), so the full taxonomy cannot be assessed from the abstract alone. No release date, dataset link, code availability or licensing information is given, so it is not yet clear whether RealSWE itself can be reproduced or reused by other researchers.