News · 2026-07-28

StateAct: agents that edit the file instead of the screenshot

A new agent design called StateAct gives computer-use agents direct code access to the files, databases and DOM underneath an application, rather than requiring them to infer everything from screenshots. On the long-horizon OSWorld 2.0 benchmark with the same Opus 4.8 backbone, the authors report roughly one-third more fully completed tasks, about seven additional partial-credit points, and an estimated ninefold lower cost per task. The critical ablation is what makes the claim interesting: swapping state-first action back to GUI action dropped performance below the screenshot baseline.

Key facts

• StateAct is a harness, not a model - the same Opus 4.8 backbone is used in the comparison.
• Reported on OSWorld 2.0: about one-third more fully completed tasks and an estimated ninefold lower per-task API cost.
• A narration-blind verifier checks the saved artifact rather than the agent's account, but rejected only about one in ten non-perfect tasks.
• Paper: arXiv:2607.22798. No repository is linked from the arXiv or Hugging Face pages.

The premise is a single observation that seems obvious once stated: a screenshot is a lossy rendering of a computer, and the thing the user actually wants changed is almost never the picture.

If you ask an agent to update a spreadsheet, the deliverable is the saved file. If you ask it to fix a configuration, the deliverable is the config on disk. If you ask it to change a record, the deliverable is a row in a database. The screen is a projection of those things, generated for human eyes - and a screenshot-driven agent is asked to reconstruct the underlying state from the projection, act on it by moving a cursor, and then read the projection again to find out whether it worked.

That is like renovating a house by watching a live camera feed and shouting instructions, when you could walk in and pick up the tools.

StateAct's main agent does the latter. It uses Bash, file editing, structured plans and delegation - no mouse, no keyboard. It probes where an application persists its state, edits that directly, and delegates the genuinely visual work to a GUI specialist. Its browser worker reads serialized DOM and clicks selectors rather than interpreting a rendered page. The overall architecture is a state-first main agent with a visual fallback, not the other way round.

The cost difference follows mechanically. Screenshots are large inputs that must be re-sent every time the screen changes, and each interaction is a full perceive-decide-click cycle. Reading a file and writing it back accomplishes the same change in a fraction of the tokens and a fraction of the steps.

But the result that carries the argument is the ablation. Replacing state-first action with GUI action, holding everything else constant, fell below the screenshot reference - and giving an agent shell access alone, without the state-first design, also underperformed. That is the useful finding, because it distinguishes the paper's claim from "we gave it more tools." Tools alone did not do it. Reorienting the agent's model of what it is acting on did.

The second component is a verification idea worth stealing regardless of what happens to the rest. Most agent self-checks ask the agent whether it succeeded, which produces exactly the answer you would expect. StateAct's finish gate is narration-blind: it sees the task and the machine but not the acting agent's story or its expected values, and then inspects the actual saved artifact. It cannot be talked into agreement, because it never hears the argument.

The authors are honest about that gate's limits. It catches structural failures - output missing, unsaved, in the wrong place, or malformed - but it rejected only about one in ten non-perfect tasks reaching it, and the paper identifies reasoning errors as the dominant remaining failure class. A file that exists, is saved, is well-formed and contains the wrong numbers sails straight through.

Two caveats matter for anyone tempted to over-read this. First, the paper is explicit that this is not "screenshots are obsolete." StateAct has little leverage on layout, image editing, WYSIWYG output, canvas work, or anything whose relevant value exists only on screen - and using a weaker GUI fallback, which was fine on several shorter and mobile tests, materially hurt the hard long-horizon desktop result. The visual channel is not vestigial; it is specialized. Second, no repository is linked from the verified arXiv or Hugging Face pages. Treat this as a paper result awaiting a release, not a package you can install.

A related paper from the same window asks a complementary question from the opposite direction. Multi-Head Latent Control (arXiv:2607.14277, code) reads the model's internal hidden-state trajectory to decide whether to continue, escalate to a stronger model, ask a question, use a tool, abstain, or just answer. It freezes a model and attaches small heads - a final-layer capability head for escalation, a middle-layer resolution head for intervention. Where StateAct is about the agent perceiving the world accurately, MHLC is about the agent perceiving itself accurately enough to know when to stop.

They would compose well in principle and cannot compose today: MHLC needs white-box access to hidden states, and StateAct's headline system runs on an API model that does not expose them.

The line worth carrying away: agents often fail not because they reason badly but because they are operating on a picture of the computer rather than the computer. Moving the action behind the glass is a cheaper fix than making the reasoning smarter.

See also: JarvisHub makes the canvas the agent's memory and agents lose the plot when you change your mind.

Primary source, verified: read the paper → (arXiv 2607.22798)

Ground Truth. (2026, July 28). StateAct: agents that edit the file instead of the screenshot. Ground Truth. https://groundtruth.day/news/stateact-agents-that-edit-the-file-instead-of-the-screenshot.html

@misc{groundtruth:stateact-agents-that-edit-the-file-instead-of-the-screenshot,
  title  = {StateAct: agents that edit the file instead of the screenshot},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/stateact-agents-that-edit-the-file-instead-of-the-screenshot.html}
}

Topics: agents · computer-use · research · tool-use · benchmarks