News · 2026-08-09

The harness, not the model, moved DeepSeek's score by twenty tasks

DeepSeek's V4 Flash 0731 release solved 44 of 89 tasks on an independent local benchmark and looked like a regression. Run the identical weights inside a different agent harness and the same benchmark returns 64 of 89 - a twenty-task swing with nothing about the model changed. The benchmarker who published both numbers, Harrison "sentdex" Kinsley, had gone on record doubting DeepSeek's own claims before his rerun vindicated them.

Key facts

• The swing: 44/89 (49.4%) on Kinsley's own harness, 64/89 (71.9%) on the Oh My Pi harness - the same model, same benchmark.
• The control: GLM-5.2 at 3.25 bits per weight moved far less over the same swap, 61/89 to 65/89.
• When and who: published this week by Harrison Kinsley, who runs the models locally on four RTX Pro 6000 cards.
• Primary source: "The Right Harness Is All You Need", Terminal-Bench v2.1, 89 tasks, 13 models.

A harness is the ordinary code wrapped around a language model in an agent: the loop that feeds it a task, hands it tools, decides what stays in its context, and decides when it is finished. It is unglamorous plumbing, and for most of the last two years the industry has treated it as a rounding error next to the weights. Kinsley's post is the cleanest public argument that it is not a rounding error at all - and it is unusually credible because he published it against his own earlier position.

The setup is worth understanding because it is not a lab result. Kinsley runs these models on hardware in his own building, benchmarking them on Terminal-Bench 2.1, a suite of 89 real terminal tasks spanning git operations, databases, reverse engineering, build systems, and machine-learning chores. His control harness is minion, a deliberately simple one he wrote. The challenger is Oh My Pi, a full-featured community harness he kept hearing about.

His first pass on DeepSeek V4 Flash 0731 was damning. "DSV4F-0731 was claiming a better-than-GLM5.2 benchmark list, which I found very tough to believe and simply could not replicate," he wrote. "I almost tossed these models into the bin for good." He suspected model makers were "overfitting to their harnesses and overfitting those harnesses to benchmarks."

Then he ran it again under Oh My Pi. The score climbed by twenty tasks. His own summary of the reversal is the most quotable thing in the post: "I went from thinking Deepseek was being very dishonest in their benchmark scores to running a test that I think is fair, using OMP for both, and finding out that yeah, it really can be just as good."

Think of it like a chef and a kitchen. Give a good cook a single burner and a dull knife and the plate that comes out says as much about the kitchen as it does about the cook. Swap in a full line - prep station, mise en place, someone plating - and the same person turns out something you would pay for. The Oh My Pi harness gives the model persistent structure across a long task: room to plan, retry, and carry state that a bare loop drops on the floor. Some models turn out to have been cooking on one burner.

The reason this matters beyond one benchmark is that it breaks a comparison everyone makes casually. Two models tested under two different harnesses are not being compared to each other. Kinsley's own words: "With a simple harness, there really is no comparison between them, but with a fully-featured harness, suddenly, they're almost identical in performance!" That is a direct challenge to every leaderboard row that does not say what code was wrapped around the model, and it lands in the same week that an agent scaffold built on the same idea reached number one on GitHub Trending.

The honest caveat is the bill, and Kinsley prices it plainly. The harness that lifts DeepSeek by twenty tasks costs about 5.3 times the tokens per solved task compared with the cheap baseline, and roughly 154,000 tokens per win against about 30,000. For GLM-5.2 the harness bought four extra tasks for more than double the time per solve. That is the trade: scaffolding converts money and latency into capability. It is not free intelligence, it is test-time compute spent well.

It also cuts the other way. The gains were wildly uneven - twenty tasks for one model, four for another - which means "the harness matters" is not a universal correction you can apply to a leaderboard after the fact. It matters differently per model, and the only way to know is to run both. That is expensive, which is precisely why almost nobody does, and why the numbers you read are usually comparing kitchens as much as cooks. It is the same finding a formal harness paper reported earlier this month, now reproduced by an independent party on his own hardware, with the receipts published against his own prior claim.

Primary source, verified: read the paper →

Ground Truth. (2026, August 9). The harness, not the model, moved DeepSeek's score by twenty tasks. Ground Truth. https://groundtruth.day/news/the-harness-not-the-model-moved-deepseek-by-twenty-tasks.html

@misc{groundtruth:the-harness-not-the-model-moved-deepseek-by-twenty-tasks,
  title  = {The harness, not the model, moved DeepSeek's score by twenty tasks},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/the-harness-not-the-model-moved-deepseek-by-twenty-tasks.html}
}

Topics: agents · harness · benchmarks · evaluation · local-llm · deepseek · open-weights