News · 2026-08-11

The new refactoring benchmark stops the best agent at 41 percent

A new coding benchmark called SWE-Bench ProMax stopped the best frontier model at a 41.2% resolve rate, far from the saturation that has made older software-engineering evaluations hard to read. Built from real commits across seven programming languages, it tests multi-file refactoring rather than localised bug fixing, and it was constructed specifically to fix the test-quality problems that an audit found in nearly 60% of its predecessor's unsolved instances.

Key facts

• The best frontier model resolves only 41.2% of tasks under two different agent scaffolds.
• 170 instances drawn from real commits across Python, Java, TypeScript, Go, C, C++ and Rust, averaging 11.4 modified files and 261.6 lines changed each.
• The motivating audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests.
• Primary source: SWE-Bench ProMax by Yuling Shi and colleagues, with the dataset on Hugging Face. Third most upvoted paper of the day.

The problem with the benchmark everybody quotes

For two years the standard way to claim a coding agent works has been a SWE-bench number. The paper opens by explaining why that number stopped meaning much: "a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data."

Both halves are damaging. Broken tests mean an agent can produce a correct fix and be marked wrong, or produce a wrong fix and be marked right. Memorised gold patches mean the score partly measures whether the answer was in the training data. Add near-saturation on top and you have a benchmark that no longer separates systems. For background, see how AI is benchmarked.

Why refactoring is the harder test

The design choice that makes ProMax interesting is the task type. Fixing a bug is usually local -- find the wrong line, change it, done. Refactoring is not. It means restructuring code across many files while changing nothing about what the program does, which requires holding the whole system's behaviour in mind while rewriting its shape.

The scale reflects that: instances average 11.4 modified files and 261.6 lines of code, well beyond typical benchmark tasks. It is the difference between replacing a cracked tile and rerouting the plumbing behind the wall so that every tap still runs exactly as it did.

That distinction connects to a failure we have reported before -- coding agents pass the tests by wrapping old code instead of deleting it. An agent that satisfies tests by accreting layers is precisely the agent a refactoring benchmark should catch.

The curation is the contribution

ProMax's real work is editorial. Issue descriptions were "rewritten from scratch to provide precise, unambiguous specifications." Test suites were "manually reviewed to remove overly narrow and overly broad tests." Tasks with insufficient complexity or limited cross-file scope were filtered out entirely.

None of that is glamorous and all of it is expensive, which is exactly why benchmarks decay. Building the dataset is a paper; maintaining the dataset is a job nobody is funded for. The multilingual spread matters too: seven languages, so a model cannot ride Python-heavy pretraining to a good score.

Why it matters

A benchmark where the best system reaches 41.2% is a benchmark with somewhere to go. That is the whole point. Evaluation only does its job when it discriminates, and for the last several months the standard coding evaluation has been compressing the entire field into a narrow band near the top -- which tells buyers, researchers and users nothing.

There is also a healthy signal in the timing. ProMax landed the same week as several papers on whether agents can improve their own scaffolding, including Evo-Bench and Ouroboros. Raising the ceiling and measuring self-improvement at the same time is how a field avoids fooling itself. Related: agent harnesses and scaffolding.

The honest caveat

170 instances is small, and small benchmarks are noisy -- a handful of tasks can move a headline percentage by several points. Hand-curation trades one bias for another: you remove contamination and bad tests, and you introduce the curators' judgement about what counts as a fair specification. And rewriting issue descriptions "from scratch to provide precise, unambiguous specifications" makes the task cleaner than the job it models. In real software work the ambiguity in the ticket is the hard part. ProMax measures whether an agent can execute a well-specified refactor. It does not measure whether an agent can figure out what you meant.

Primary source, verified: read the paper → (arXiv 2608.09802)

Ground Truth. (2026, August 11). The new refactoring benchmark stops the best agent at 41 percent. Ground Truth. https://groundtruth.day/news/the-new-refactoring-benchmark-stops-the-best-agent-at-41-percent.html

@misc{groundtruth:the-new-refactoring-benchmark-stops-the-best-agent-at-41-percent,
  title  = {The new refactoring benchmark stops the best agent at 41 percent},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/the-new-refactoring-benchmark-stops-the-best-agent-at-41-percent.html}
}

Topics: benchmarks · coding-agents · software-engineering · evaluation · refactoring