News · 2026-08-07

A task factory ran fifteen rounds and broke the model grading it

A team led by Zhongzhi Li has published a method that manufactures long-horizon command-line tasks for AI agents by recursively rewriting tasks it has already validated, and the resulting difficulty curve is steep enough to flatten a frontier model. Across fifteen rounds, DeepSeek-V4-Pro's success rate on the generated tasks fell from 90 percent to 2.5 percent, and the authors report that "after 15 rounds, the recursion shows no ceiling." The full task set, the agent trajectories, and three fine-tuned checkpoints have been released publicly.

Key facts

• The pipeline produced 37,484 tasks at roughly five cents each, against a stated human-authoring cost of hundreds to thousands of dollars per task.
• A fixed solver's pass rate collapsed from 90 percent in round one to 2.5 percent in round fifteen.
• Posted to arXiv on August 5, 2026; it was the top-voted paper on Hugging Face's daily board for August 6.
• Primary source: Recursive Synthesis for Long-Horizon Terminal Tasks (arXiv 2608.05466).

The bottleneck this attacks is boring and expensive. To train an agent that can work in a terminal for hours, you need tasks where four things stay consistent with each other: the instruction the agent reads, the environment it runs in, a reference solution that actually works, and a verifier that can tell whether the agent's answer is right. Humans writing those by hand is slow. Asking a language model to invent one from scratch tends to break the links between the four - the instruction describes something the verifier does not check, or the reference solution does not run in the environment shipped with it.

The method, which the authors call Recursive Synthetic Terminal Tasks, sidesteps that by never starting from scratch after the first step. It begins with a verified seed task, extends the reference solution by one more stage of work, then rewrites the verifier and the instruction to match the new solution, and revalidates the whole bundle in a clean sandbox. If it survives, that task becomes a seed for the next round. Think of it less like writing exam questions and more like a codebase accreting features: each commit has to keep the tests passing, and the tests grow with it.

What that produces is not longer prompts. It is more executable work. Over the fifteen rounds the median reference solution grew from 67 lines to 374, the median number of shell commands from 40 to 244, and verifier assertions from 17 to 57 - while the instruction itself only grew from about 85 words to 122. The task got harder without getting wordier, which is the opposite of how most benchmark inflation works. The paper's clearest worked example starts as a task about running a JSON-diff tool over fixed fixtures and ends, fifteen rounds later, as one where the agent has to repair inconsistent configuration and fixture data, regenerate outputs, reconcile expected against observed counts, and pass a test suite it cannot read.

The difficulty curve is the headline result, and it is worth being precise about what it measures. The solver used to probe difficulty stays the same model throughout, so the collapse from 90 percent to 2.5 percent is a property of the tasks, not of a degrading agent. The paper also notes that failures change character: early on, failed attempts are near misses that earn substantial partial credit, and by the end they are failures that get almost nothing. The agent stops nearly-solving and starts not-solving.

Then the payoff. The authors collect agent trajectories on the synthesized tasks and fine-tune on them, reporting gains of up to ten points for Qwen3.5-27B and Qwen3.5-122B-A10B across three terminal-agent benchmarks, with a further lift from reinforcement learning on top. The artifacts are all public: a 37,484-task dataset, a 327,000-trajectory companion set, and three checkpoints - 27B supervised, 122B supervised, and 27B reinforcement-trained - gathered in a single Hugging Face collection. There is also a browsable audit view that lets you walk an individual task's lineage round by round.

The caveat matters, and the paper is honest about it. This is not a system that rewrites itself. The recursion lives entirely in the data pipeline; the models it trains are ordinary fine-tunes, and the paper reports that the stronger fixed solver still beats the trained checkpoints on the hardest benchmarks. Anyone reading this as evidence of recursive self-improvement in deployed agents is reading something that is not there. What it is evidence for is narrower and, for people building agents, more immediately useful: the supply of hard, verifiable training tasks may not be the constraint everyone assumed it was.

That theme is not confined to this one paper. The same week's board carried AgentOPSD, which recursively updates a belief state to assign credit to individual turns; ABSeeker, which works backward from a known answer to recover the clues a search should have found; EnvACE, which alternates acting with rehearsing the environment; GDPevo, on attributable self-evolution over real business workflows; CalibForge, which calibrates task difficulty against solver behavior rather than mere solvability; and a survey of self-evolving coding agents. Different mechanisms, one shared instinct: stop tuning the agent, start rewriting what it learns from. It is the same instinct behind the field's turn toward verifiable rewards and synthetic training data, pushed one level further back.

Primary source, verified: read the paper → (arXiv 2608.05466)

Ground Truth. (2026, August 7). A task factory ran fifteen rounds and broke the model grading it. Ground Truth. https://groundtruth.day/news/the-task-factory-that-broke-its-own-solver.html

@misc{groundtruth:the-task-factory-that-broke-its-own-solver,
  title  = {A task factory ran fifteen rounds and broke the model grading it},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/the-task-factory-that-broke-its-own-solver.html}
}

Topics: research · agents · synthetic-data · training · evaluation · open-weight-models · reinforcement-learning