News · 2026-08-07

The AI judges grading computer-use agents are too easy on them

A benchmark released by Qiushi Sun and a large collaborating team finds that the vision-language models used to grade computer-use agents are not reliable enough for the job, and that their dominant failure is leniency: accepting runs the agent did not actually complete. The team frames it as a question nobody had asked out loud. "A fundamental question has long gone unexamined," they write: "are these VLM judges reliable enough?"

Key facts

• The benchmark, OSReward, grades judges against human-annotated ground-truth verdicts on agent trajectories collected across web, Windows, Ubuntu, and mobile.
• The paper reports that even state-of-the-art models fall short of an ideal judge, and that agreement varies more by platform than by which judge is used, with desktop the hardest.
• Submitted July 30, 2026, revised August 6; the team also released OS-Shepherd reward models at 9B and 35B trained to cut false successes.
• Primary source: OSReward (arXiv 2607.28609); project page, code.

Here is the setup. A computer-use agent is a model that operates a real machine - clicking, typing, running commands - and its record of what it did is called a trajectory: the actions it took, the screen states it saw, and its own reasoning along the way. Somebody has to decide whether that trajectory actually completed the task. Hand-written checkers work only for tasks somebody anticipated. Humans are accurate and far too slow. So the field defaulted to pointing another model at the trajectory and asking it to score. That judgment then feeds evaluation numbers, training-data filtering, and reinforcement learning - which means an error in the judge does not stay in the judge. It propagates into what the next agent learns to do.

OSReward is built to test that layer directly. The team collected trajectories from a range of agent backbones executing human-verified instructions across four platforms, then labeled each with a ground-truth verdict through multi-stage human annotation. On top of the main set they derive OSReward-Hard, a challenge subset concentrating genuinely difficult cases, and OSReward-Multi for finer-grained scoring of efficiency and alignment. It is, as the paper describes it, the most comprehensive evaluation of these judges to date.

The headline finding is not that the judges are noisy. Noise averages out. The finding is that they are biased in one direction - they say pass when the answer is fail. That asymmetry is what makes it dangerous rather than merely annoying. A judge that errs both ways adds variance to a leaderboard. A judge that only errs generously adds a systematic upward tilt to every number it touches, and quietly teaches agents that stopping early is fine.

The more interesting result for anyone building these systems is that judgment quality does not port across platforms. The paper reports that agreement with programmatic verifiers varies far more by operating system than by which judge you pick, with mobile the easiest and desktop the worst. That has a plain-language explanation: on a phone, the screen usually shows you whether the thing happened. On a desktop, the evidence of success is often not on screen at all - it is a file that was written, a service that restarted, a state that changed somewhere the screenshot cannot see. A judge reading pixels is reading the wrong surface. This is a cousin of the finding that the same model scores 52 or 81 percent depending on the harness around it: the measurement apparatus is doing more work than anyone admits.

The team does not stop at diagnosis. They release OS-Shepherd-100K as training data and two reward models, OS-Shepherd-9B and a 35B sibling, trained specifically to reduce false-success verdicts, along with the benchmark dataset and code. That makes this a usable artifact rather than a complaint - if you are running a computer-use pipeline today and grading it with a general-purpose model, there is now a purpose-built alternative to compare against.

The caveat is that a benchmark for judges is itself a judged artifact. The ground truth here comes from human annotators, and the paper's difficulty subset is constructed by the same team that built the main set, so the ceiling it defines is theirs. And several of the sharper specifics circulating about this paper - exact scores on the hard set, the precise size of the platform gap - are drawn from the full text rather than the abstract, and are worth reading in the paper rather than taking secondhand.

The broader point stands regardless, and it is the one this site keeps arriving at from different directions: model judges are reliable enough to be trusted and wrong often enough to be dangerous. Every layer of the modern training stack - data filtering, preference learning, reward modeling - now has a model somewhere in it deciding what counts as good. The LLM-as-a-judge pattern was adopted because it scaled, not because anyone had established it was accurate. This paper is what checking looks like.

Primary source, verified: read the paper → (arXiv 2607.28609)

Ground Truth. (2026, August 7). The AI judges grading computer-use agents are too easy on them. Ground Truth. https://groundtruth.day/news/nobody-had-checked-whether-the-ai-judges-of-computer-agents-are-right.html

@misc{groundtruth:nobody-had-checked-whether-the-ai-judges-of-computer-agents-are-right,
  title  = {The AI judges grading computer-use agents are too easy on them},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/nobody-had-checked-whether-the-ai-judges-of-computer-agents-are-right.html}
}

Topics: research · agents · evaluation · llm-as-a-judge · computer-use · reward-models · benchmarks