News · 2026-08-22

Coding agents ace the public test and stumble on the hidden one

A coding agent can pass 96.6 percent of the visible tests on a scientific software task and still fail more than half the tasks outright. That is the gap SWE-bench Science was built to expose, and the official leaderboard now shows it plainly: the top entry clears 96.6 percent of public checks, 75.1 percent of private checks, and finishes only 47.9 percent of tasks on the first attempt. The benchmark holds 119 tasks pulled from 98 GitHub repositories across 20 scientific domains, and it keeps the grading tests out of the public release entirely.

Key facts

• 119 tasks from 98 repositories across 20 scientific domains, split into 52 issue-driven, 49 expert-exploratory, and 18 engineering-integration tasks.
• The top leaderboard entry scores 96.6 percent public, 75.1 percent private, and 47.9 percent first-attempt success.
• Submitted August 20, 2026 by the OpenMOSS group as arXiv 2608.19799, with a public repository and live leaderboard.
• The public release contains no reference patch, no private tests, no grader, and no agent trajectories.

The design decision that matters is the split into two container images. One is the environment: the code, the dependencies, a diagnostic check the agent can run while it works. The other is the verifier, which holds the real tests and the grader, and it only runs after submission. Nobody building against this benchmark can see what they will be graded on. That is a direct response to how the original SWE-bench Verified was critiqued, including OpenAI's own writeup noting overly specific tests, underspecified tasks, and flaky environments, and Ground Truth's coverage of the finding that one in seven SWE-bench Verified tasks is graded against a patch that does not match.

What makes the science framing more than a label is what the hidden tests check. The paper gives examples: FFT mesh parity, k-point ordering, cell geometry, and MDF integrals. These are not code-style assertions. They are physical and mathematical invariants that the software must preserve to remain correct, and they are exactly the sort of thing you can violate while every visible test stays green. A patch that makes the failing case pass but silently breaks the ordering of points in reciprocal space is, to a normal test suite, a success. To a physicist it is a bug that will poison every result downstream.

That is why the public-private gap is the whole story. Think of it as the difference between a student who memorized the practice exam and one who understood the material. On the practice questions they look identical. Change the numbers and only one of them survives. The visible diagnostic check tells an agent whether the symptom went away; the hidden verifier asks whether the fix was actually right. Agents are demonstrably very good at making symptoms go away, which connects to shortcut learning and to the broader problem of how AI gets benchmarked.

The paper's failure audit names four recurring patterns, and they are more diagnostic than a score. Agents lack the domain knowledge or the right abstraction for the problem. They explore in the wrong direction or repair only the surface. They fix one call site and miss the rest of the system. And they fail to generalize a scientific principle beyond the specific case they observed. Every one of those is a knowledge failure rather than a coding failure, which is the point: writing Python was never the hard part of scientific software.

One ablation deserves attention because it complicates the obvious fix. The authors ran 91 tasks with extra scientific information supplied to the agent. For one model, that information slightly raised both public and private scores while lowering first-attempt success and reducing token use. For another, it improved all three but cost more tokens. The reading offered is that scientific context helps when it is well grounded and hurts when it anchors the agent to the wrong frame. So you cannot simply hand the model a textbook and expect the gap to close; badly matched context makes a confident agent confidently wrong.

The honest caveat is that this is a first-party evaluation of a first-party benchmark, and the defensive architecture is a design claim rather than an audited one. Held-out tests are only as good as the discipline maintaining them, and every hidden benchmark eventually leaks as submissions accumulate and results are discussed. The Hugging Face paper page shows modest early attention. Still, the direction is right, and the number worth remembering is the one nobody wants on a slide: the same system that passes 97 percent of what you can see finishes fewer than half the jobs.

Primary source, verified: read the paper → (arXiv 2608.19799)

Ground Truth. (2026, August 22). Coding agents ace the public test and stumble on the hidden one. Ground Truth. https://groundtruth.day/news/coding-agents-ace-the-public-test-and-stumble-on-the-hidden-one.html

@misc{groundtruth:coding-agents-ace-the-public-test-and-stumble-on-the-hidden-one,
  title  = {Coding agents ace the public test and stumble on the hidden one},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/coding-agents-ace-the-public-test-and-stumble-on-the-hidden-one.html}
}

Topics: benchmarks · coding-agents · science · evaluation · research