News · 2026-09-13

Real-SWE tests coding agents on private company codebases, and the best solved fewer than four in ten tasks

Specific Labs published Real-SWE on 12 September 2026, a benchmark that tests AI coding agents on tasks taken from private production codebases licensed from real companies, and the best agent solved them less than 40% of the time. Claude Fable 5.1 ranked first and GPT-6 Astra second, but the headline is how far every model fell short of reliable enterprise work.

Key facts

• Anchor number: the top configuration, Claude Fable 5.1 in Claude Code, resolved 38.8% of attempts; six of the ten tasks were solved less than 15% of the time.
• Scale: “Eight model and harness configurations, ten tasks, 640 scored rollouts,” with each task run eight times per model.
• Who: Specific Labs, a company that turns company workflows into training and evaluation data for AI.
• Primary source: the Real-SWE results page; discussion on Hacker News.

Why a private benchmark

Public coding benchmarks are built from open-source repositories, and open-source code is exactly what large models are trained on. That creates a risk of benchmark contamination: a model can look skilled on a test partly because it has seen the material before. Ground Truth reported last week that a verified rebuild of one popular coding benchmark found models had scored too high.

Specific's pitch is that most real code is nothing like that public pool. “99% of tokens in real-world enterprises are hidden away from the frontier models,” the page says. “Each task comes from a private production codebase that we licensed from a real-world company.” Examples it names include a consumer fintech platform processing more than 100,000 bank statements and an events app with more than 200,000 users.

How it works

Each model ran inside a coding agent harness, such as Claude Code, Codex CLI or Gemini CLI, in an isolated sandbox. Hidden tests, adapted from or copied from each company's own test suites, were added only at grading time. A task counts as resolved if those tests pass, and the published rate averages eight independent attempts per task.

A useful comparison is a new hire's first week. Solving a textbook exercise proves general skill. Shipping a change in a company's own codebase, with its conventions, half-documented requirements and existing tests, is a different job, and that is what Real-SWE tries to measure. The reference fixes touched a median of eleven files, roughly twice as many as some well-known public coding tests.

What they found

The ranking put Claude Fable 5.1 first at 38.8%, GPT-6 Astra second at 33.8% and Gemini 3.8 Flash third, with the Chinese open model GLM 5.3 close behind. Kimi K3 and the older GPT-5.6 Sol trailed. One task, an analytics stream reducer, was solved by no model at all.

The failure pattern was consistent: the most common reason for a failed attempt was a missed requirement. “We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions,” the authors write. Their conclusion is blunt: “Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality.” Each attempt cost between $2.50 and $6.96 in model usage.

Name the incentive

Specific is not a neutral referee. Its about page says it “turns real workflows into datasets for dependable agents” and works with frontier labs, and a separate page tells businesses their company data “could be worth” $100,000 to $1 million to AI developers. A benchmark showing that models struggle on private code is also an advertisement for private code as training data. That does not make the results wrong, but it is a reason to want them checked.

The pushback

On Hacker News, practitioners zeroed in on reproducibility. “So basically pinky-promise benchmarking?” wrote one commenter. Another, who runs a separate benchmark, warned that many large “private” codebases are “not in fact private at this point” because developers have fed them to coding assistants, and urged measuring contamination every time. A co-founder replied that the company is “going to open source some of our tasks and model trajectories as well.” For more on why leaderboards mislead, see our lesson on how AI is benchmarked.

The caveat

Ten tasks is a very small sample, and a single task can swing a model's rank. The companies, repositories and tasks cannot be inspected, the page contradicts itself in one footnote about how many tasks are repository-backed, and no human baseline exists to show how hard the tasks are for a new engineer. Treat the ordering as suggestive and the overall message, that agents still miss requirements in unfamiliar code, as the durable finding.

Primary source, verified: read the paper →

Ground Truth. (2026, September 13). Real-SWE tests coding agents on private company codebases, and the best solved fewer than four in ten tasks. Ground Truth. https://groundtruth.day/news/real-swe-tests-coding-agents-on-licensed-private-codebases-and-the-best-solves-under-four-in-ten.html

@misc{groundtruth:real-swe-tests-coding-agents-on-licensed-private-codebases-and-the-best-solves-under-four-in-ten,
  title  = {Real-SWE tests coding agents on private company codebases, and the best solved fewer than four in ten tasks},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/news/real-swe-tests-coding-agents-on-licensed-private-codebases-and-the-best-solves-under-four-in-ten.html}
}

Topics: benchmarks · coding-agents · evaluation · anthropic · openai · enterprise