News · 2026-09-03

AI agents built 18 versions of their own infrastructure and not one ever saved its work

A benchmark called HarnessDev had six frontier models build and then iteratively improve their own agent infrastructure, and found a specific, damning gap. All 18 generated code harnesses implemented an execution loop. Only 11 defined a state class, only one exposed state saving, and only one checkpointed periodically -- and across 26,679 recorded agent trajectories, not a single checkpoint event ever occurred.

Key facts

• Across 26,679 recorded trajectories, zero checkpoint events. Of 18 code harnesses, all implemented an execution loop; only one checkpointed periodically.
• Scope: six creator models, four domains, five downstream benchmarks, 2,207 unique downstream task instances.
• Self-improvement generalises poorly: across 64 comparable switches, visible feedback and held-out performance moved in the same direction only 53.1% of the time.
• Primary sources: the HarnessDev paper and its project page.

The paper's framing is the useful part even before the results. It treats the harness -- not the model -- as the object of evaluation, and defines it as six components: execution loop, tool policy, context management, state and memory, lifecycle and recovery, and result verification. Almost every public agent comparison holds the harness fixed and varies the model, which quietly assumes the harness is neutral plumbing. It is not, and our explainer on agent harnesses and scaffolding covers why the wrapper often matters more than the weights.

HarnessDev runs two stages. Creation starts each model from a deliberately crippled seed -- runnable, but with no loop, no planner, no verifier -- and asks it to build a working harness. Evolution starts from the model's own harness and has it revise using feedback from actual downstream execution. Everything is measured against verified public human-engineered harness and executor pairs.

The Creation results split by domain in a way that makes sense once you see it. Machine-built harnesses still lag mature human systems in code, search and research -- domains where decades of tooling conventions encode hard-won knowledge. But they match or exceed the human references in writing and machine-learning experimentation, where the conventions are thinner and the task is more self-contained.

Then there is the checkpoint finding, which is the number worth remembering. Every harness knew how to run. Almost none knew how to survive. In practice that means an agent working a long task that hits an API error, a rate limit or a provider outage -- the kind three major labs all had on September 3 -- loses everything and starts over. Think of a builder who frames a house beautifully and never installs a door: the structure is right, and the first time anyone needs to get out, it is useless.

The reason is not stupidity, it is incentive. Checkpointing has no reward signal. It never makes a benchmark score go up. It only prevents a catastrophe that the benchmark does not measure, which means a model optimising against visible feedback has no reason to build it. That is reward hacking in its most mundane and most instructive form -- not a model cheating, just a model correctly ignoring what it was not asked about.

The cost findings puncture another assumption. The paper reports roughly nineteen-fold variation in execution-token use on one machine-learning benchmark, and the expensive runs do not reliably score better. Edit size is equally uninformative: the 18 code artifacts add 17,111 net lines in total, but the creator that adds the fewest lines takes the best score on one terminal benchmark. Self-test count is a weak predictor; revision calls correlate much better. More code, more spending and more tests all fail as proxies for quality, which is inconvenient for basically every dashboard measuring agent work today.

The Evolution results are the sobering ones for anyone excited about self-improving systems. All five self-runtime lineages improve on the feedback they can see. Held-out gains are consistently smaller. Under a fixed executor, only one creator improves on held-out tasks while three actually regress. And the 53.1% figure -- visible feedback and held-out performance agreeing barely more often than a coin flip -- means an agent watching its own metrics improve has close to no information about whether it is genuinely getting better. Only 2 of 9 declared final harnesses were the held-out optimal choice.

There are real successes in the transcript. The best is a model noticing that 99 of 100 runs reported success while only 48 actually passed, then adding a completion check to catch the discrepancy -- exactly the kind of verification gap a careful engineer would find. The worst pattern is the mirror image: executor-specific logic, hard-coded limits and sanitizers tuned to one runtime, which shatter the moment the runtime changes. Optimising hard against a fixed environment produces something that only works in that environment.

The paper is well received -- #2 Paper of the day on Hugging Face with 225 upvotes -- and it extends earlier work Ground Truth covered on models rewriting their own harness, by testing from-scratch construction, iterative evolution, and portability across executors rather than a single rewrite outcome.

The honest caveat: these are harnesses built by models under time and budget constraints, compared against human systems refined over years by teams with production incentives. That comparison is unfair by construction, and the paper says so. What survives the unfairness is the structural finding -- machines building infrastructure build the parts that get measured and skip the parts that only matter when something goes wrong.

Primary source, verified: read the paper → (arXiv 2609.01437)

Ground Truth. (2026, September 3). AI agents built 18 versions of their own infrastructure and not one ever saved its work. Ground Truth. https://groundtruth.day/news/agents-that-build-their-own-harness-never-once-saved-state.html

@misc{groundtruth:agents-that-build-their-own-harness-never-once-saved-state,
  title  = {AI agents built 18 versions of their own infrastructure and not one ever saved its work},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/news/agents-that-build-their-own-harness-never-once-saved-state.html}
}

Topics: agents · benchmarks · agent-harnesses · research · evaluation · self-improvement