News · 2026-07-31

13.6% of SWE-bench Verified pairs a bug report with a patch that does not match it

SWE-bench Verified is the benchmark that most claims about AI coding ability rest on, and a systematic audit published July 30 found that 13.6% of its instances pair a bug report with a code change that does not match it. In a 500-task benchmark, that is 68 tasks where the problem statement and the grading oracle describe different work - across five distinct failure patterns and eleven finer-grained scenarios.

Key facts

• 13.6% of SWE-bench Verified instances exhibit pull-request-to-issue misalignment, across five patterns.
• The authors' detector, PAIChecker, reaches up to 92.12% binary accuracy on SWE-Gym and 91.67% on SWE-bench Multilingual.
• Published July 30, 2026, by Manyi Wang, Junjielong Xu and Pinjia He.
• Primary source: PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks.

The defect is structural, not sloppy. SWE-bench-style benchmarks are built by an automated pipeline: scan a repository's merged pull requests, extract any issue number mentioned in the description, use that issue's text as the problem statement, and use the pull request's tests as the answer key. It is an elegant way to harvest thousands of realistic programming tasks for free. It also inherits every habit real developers have.

Real developers reference an issue in a pull request for many reasons that are not "this patch fixes that issue." They fix the bug and refactor three neighbouring functions in the same commit. They fix part of a report and leave the rest for later. They mention a related ticket for context. They fix a different problem they noticed while reading the report. Each of those is normal engineering practice, and each produces a benchmark task whose stated problem and whose grading oracle have drifted apart.

The consequence for how models get benchmarked is specific. On a misaligned task, an agent that reads the issue and correctly fixes exactly what was reported can still fail the hidden tests, because the tests check the refactor that came along for the ride. Conversely, an agent that happens to produce the broader change can pass without having understood the report. The noise runs both ways, which is worse than a uniform bias - it cannot be subtracted out, and it does not affect all models equally.

The authors' response is a detector rather than a cleanup. PAIChecker is a three-phase multi-agent system: identify which of the known misalignment patterns a candidate matches, synthesise a label across several agents, then validate at the code level rather than trusting the description. On SWE-Gym and SWE-bench Multilingual it reaches roughly 92% binary accuracy across four different language-model backbones, which is the useful part - it is meant to be run as a filter when building the next benchmark, not just to audit the current one.

The five patterns the authors identify are worth knowing by name, because each produces a differently broken task. A patch can address only part of what the issue reported; it can address the issue plus a substantial amount of unrelated work; it can fix a different problem entirely while merely referencing the issue; the issue itself can be too vague to specify any particular fix; or the linkage can be incidental, a developer citing context rather than declaring a fix. Eleven finer-grained scenarios sit under those five. None of them is a mistake by the benchmark's authors - they are the ordinary texture of how software gets maintained, harvested by a pipeline that assumed otherwise.

This lands in a week when benchmark hygiene keeps surfacing. The official MLE-bench repository has paused new leaderboard submissions over comparability concerns, and a separate July 30 paper, Change2Task, builds executable coding tasks from repository history with explicit lifecycle validation - reporting 79.6% verified construction success and 29.2% more verified tasks than a pull-request-based baseline. Different teams, same underlying worry: the supply of automatically harvested agent tasks has outrun the checking of it.

The honest limit is that this study documents a defect rate, not a corrected leaderboard. It does not re-score existing models on a cleaned subset, so nobody yet knows whether removing those 68 tasks shuffles the rankings a little or a lot. The authors show that removing misaligned instances can change ranks; how much, for which models, remains open. What is settled is that a benchmark quoted to one decimal place has roughly one task in seven where the question and the answer key disagree.

Primary source, verified: read the paper → (arXiv 2607.28587)

Ground Truth. (2026, July 31). 13.6% of SWE-bench Verified pairs a bug report with a patch that does not match it. Ground Truth. https://groundtruth.day/news/one-in-seven-swe-bench-verified-tasks-is-graded-against-a-patch-that-does-not-match.html

@misc{groundtruth:one-in-seven-swe-bench-verified-tasks-is-graded-against-a-patch-that-does-not-match,
  title  = {13.6% of SWE-bench Verified pairs a bug report with a patch that does not match it},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/one-in-seven-swe-bench-verified-tasks-is-graded-against-a-patch-that-does-not-match.html}
}

Topics: benchmarks · coding · evaluation · swe-bench · agents · research