SWE-Bench ProMax caps the best coding agent at a 41.2% resolve rate
Researchers introduced SWE-Bench ProMax, an expert-curated, multilingual benchmark for testing AI coding agents on large-scale code refactoring. It was built as a response to quality problems found in existing agent benchmarks: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, either too narrow to accept correct solutions or too broad to check requirements the issue never stated, and that frontier models can reproduce gold patches from training data verbatim rather than actually solving the task. The authors argue that code refactoring, which requires coordinated, behavior-preserving changes across many files at once, is a substantially harder and more realistic test of agent capability than existing benchmarks provide, yet has been underserved by them. SWE-Bench ProMax consists of 170 instances drawn from real commits across seven programming languages: Python, Java, TypeScript, Go, C, C++, and Rust. Every instance went through multi-stage curation aimed directly at the problems identified above: issue descriptions were rewritten from scratch for precise, unambiguous specifications, and test suites were manually reviewed to strip out overly narrow and overly broad tests. Tasks judged too simple or too limited in cross-file scope were filtered out entirely. The result averages 11.4 modified files and 261.6 lines of code changed per instance, well beyond the scale of prior benchmarks. When frontier models were run against the benchmark under two different agent scaffolds, the best-performing model resolved only 41.2% of instances, which the authors take as confirmation that SWE-Bench ProMax remains a meaningful, unsaturated challenge for current AI coding agents. The source text does not name which models or scaffolds were tested, nor does it identify the audit of SWE-bench Verified it cites or give a release timeline for the benchmark. The dataset itself is public, hosted on Hugging Face.
Key facts
- SWE-Bench ProMax is a 170-instance, expert-curated refactoring benchmark spanning seven languages: Python, Java, TypeScript, Go, C, C++, and Rust.
- A cited audit found nearly 60% of unsolved SWE-bench Verified instances contain flawed tests, and that frontier models can reproduce gold patches from training data verbatim.
- Each instance averages 11.4 modified files and 261.6 lines of code, far larger in scope than prior code-agent benchmarks.
- Tested under two agent scaffolds, the best frontier model resolved only 41.2% of instances.
- The benchmark is publicly available on Hugging Face at swe-bench-promax/SWE-Bench-ProMax.
Why it matters
Popular agent benchmarks like SWE-bench Verified are showing their age: a cited audit found flawed tests in nearly 60% of unsolved instances, and frontier models have been caught reproducing memorized gold patches rather than solving problems from scratch. SWE-Bench ProMax targets both failures directly, and picks refactoring specifically because coordinated, behavior-preserving edits across many files are a harder and more realistic proxy for real engineering work than the single-file bug fixes many older benchmarks favor.
Who it affects
Teams building and evaluating AI coding agents get a benchmark less prone to the false positives and memorization shortcuts that inflated scores on older tests. Researchers studying multi-file, cross-language reasoning in agents gain a harder target with headroom left to measure against, since the best model tested still fails on the majority of tasks.
How to use it
The benchmark is public on Hugging Face at huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax, with 170 instances covering seven programming languages. Anyone evaluating a coding agent's refactoring ability can run it directly against the published tasks.
How solid is it
Every instance went through multi-stage expert curation: issue descriptions were rewritten from scratch for precise specifications, and test suites were manually reviewed to remove tests that were either too narrow to accept correct fixes or too broad to check only what the issue actually asked for. Tasks with insufficient complexity or limited cross-file scope were filtered out, leaving instances that average 11.4 modified files and 261.6 lines of code, a substantially larger scope than prior benchmarks.
Risks and caveats
The source text does not name which frontier models or which two agent scaffolds were tested, only that the best model reached a 41.2% resolve rate. It also does not identify the specific audit of SWE-bench Verified it cites, nor does it give a release date or timeline for the benchmark. With 170 instances total, the benchmark is a meaningful but not exhaustive slice of real-world refactoring work.