Learn · Beginner
The harness: the code around a model that decides how smart it looks
An agent harness is the ordinary software wrapped around a language model that decides what the model sees, what it can do, and what it remembers. It is not a model, not a prompt, and not usually a research contribution -- it is a loop -- and yet it can change a benchmark score by a factor of three. In July 2026 OpenAI reported that enabling two memory-related API settings took the same model from 13.3% to 38.3% on the ARC-AGI-3 puzzle benchmark while spending six times fewer output tokens. The weights never changed. Only the harness did.
What the harness actually does
A language model does exactly one thing: given a block of text, produce the next piece of text. Everything else you associate with an AI agent is the harness. It holds the loop, and on every turn it makes several consequential decisions.
It assembles the context: the system instructions, the task, the relevant history, tool descriptions, retrieved files. It executes tools: when the model emits a request to read a file or run a command, the harness actually runs it, captures the output, and truncates it if it is too long. It manages memory: as the conversation grows past the context window, something has to go, and the harness picks -- drop the oldest turns, summarize them, or keep a compressed state. And it controls the model's own reasoning trace: after each step, does the model's private thinking stay in the history, or get thrown away?
That last choice sounds administrative and is not. If a harness discards reasoning after every action, the model has to rebuild its understanding of the problem from scratch on every single turn. It can see what it did but not why. Imagine solving a Rubik's cube where after each twist your memory of your plan is wiped, though you keep a list of the twists you made. You would look far worse at cubes than you are. That was precisely the situation OpenAI diagnosed on ARC-AGI-3, alongside a truncation rule that eventually deleted the model's earliest observations too.
Why this became the central engineering problem
When models were mostly used for single answers, the harness barely mattered. A long-running agent changes that, because errors and losses compound across dozens of steps. Research on the model side has been converging on the same point for years: ReAct showed that interleaving reasoning with actions beats either alone, Reflexion showed that letting a model write and re-read its own critiques improves later attempts, and SWE-agent made the argument explicitly by naming its contribution the agent-computer interface -- designing the commands and feedback formats a model works with, not the model.
The same insight shows up in production tooling constantly. A coding agent that re-greps a repository every session is paying repeatedly for discovery a harness could cache, which is the argument behind work on serving repository context. Capping tool output so a single verbose command cannot swallow the context window is a harness decision. Loading tool descriptions only when they might be needed, rather than listing hundreds up front, is a harness decision. So is preserving an exact prompt prefix so that prompt caching keeps working, and so is how much test-time compute each step is allowed to spend.
What this means for reading benchmark numbers
Here is the uncomfortable consequence. When you see "model X scores Y on benchmark Z," you are reading a result produced by a bundle: a model, an effort or reasoning setting, a harness, a memory policy, a tool set, and sometimes a safety route that can silently hand the request to a different model. Change any element and the number moves.
Benchmark maintainers know this, and their usual response is a deliberately plain harness -- no tools, no vendor-specific features -- on the reasoning that a simple harness exposes model weaknesses and keeps comparisons fair. Vendors object that this understates what their deployed products do, since a real ChatGPT or Codex session runs with memory features switched on. Both positions are defensible, which is exactly why this is unresolved. ARC Prize has documented cases where a hand-tuned harness lifted a model dramatically on one environment and did nothing on another -- evidence that harness gains do not automatically mean capability gains.
The practical takeaways are three. If you are building on a model, the harness is your highest-leverage surface: most teams should tune memory policy, tool surface and context assembly before reaching for a bigger model. If you are comparing models, ask which harness and which settings before you believe a gap. And if you are reporting results, publish the harness -- because a score without one is not a measurement of a model. Our guides to how AI is benchmarked, agent memory and tool use cover the moving parts in more depth.
ReAct: Synergizing Reasoning and Acting in Language Models (Yao et al., 2022)
Toolformer: Language Models Can Teach Themselves to Use Tools (Schick et al., 2023)
Reflexion: Language Agents with Verbal Reinforcement Learning (Shinn et al., 2023)
SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (Jimenez et al., 2023)
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering (Yang et al., 2024)
Key questions
What is an agent harness?
Why does the harness change benchmark scores so much?
Should a benchmark use one generic harness or each vendor's own?
Cite this
APA
Ground Truth. (2026, July 29). The harness: the code around a model that decides how smart it looks. Ground Truth. https://groundtruth.day/learn/agent-harnesses-and-scaffolding.html
BibTeX
@misc{groundtruth:agent-harnesses-and-scaffolding,
title = {The harness: the code around a model that decides how smart it looks},
author = {{Ground Truth}},
year = {2026},
month = {jul},
url = {https://groundtruth.day/learn/agent-harnesses-and-scaffolding.html}
}