swe-bench
Coding Agents Pass the Tests by Wrapping the Old Code Instead of Deleting It News
A new study finds that 29% of the coding-agent patches that pass SWE-bench Verified keep code the human developer removed, usually by wrapping it in a guard or fallback, and that adding checks for the deletion drops resolution rates from 63.2% to 41.9%.
13.6% of SWE-bench Verified pairs a bug report with a patch that does not match it News
A systematic audit of SWE-bench Verified, the benchmark used everywhere to rank AI coding ability, found that 68 of its 500 tasks link an issue to a pull request that fixes something else, adds unrelated work, or only partly addresses the report.
OpenAI says a leading coding benchmark can no longer tell the best models apart News
OpenAI published an analysis concluding that SWE-Bench Pro, a widely-cited coding benchmark, has hit a roughly 70% noise ceiling where higher scores may reflect quirks rather than real skill, and retracted its recommendation to use the benchmark to rank frontier models.
'Dockerless' verifies AI code patches by reading the repo instead of running it News
A new method called Dockerless judges whether an AI's code patch is correct by having an agent explore the repository for evidence rather than executing tests in a Docker container, enabling a fully environment-free training pipeline for coding agents.