Ground Truth.
AI, checked against the source.

News · 2026-08-21

The ARC-AGI-3 record going around is the wrong number and the wrong system

The highest ARC-AGI-3 result on ARC Prize's community leaderboard is an NVIDIA-labelled agent called NOOA at 85.1% on the public demo set, spanning 25 environments, at a compute cost of $332. It is self-reported, and ARC Prize's own policy says results outside its semi-private testing are not independently verified. It is also not the system that went around this week credited with a flawless 183-out-of-183 run. That claim conflated NOOA with AVO, an unrelated NVIDIA paper about optimizing GPU kernels.

Key facts

This is a small correction with a large lesson attached, and the lesson is about a specific way benchmark news goes wrong.

Background on why ARC-AGI-3 is interesting in the first place. The earlier ARC benchmarks handed a model a set of coloured grids and asked it to infer the transformation rule. Hard, but static. ARC-AGI-3 changed the shape of the problem: agents are dropped into novel interactive environments where no instructions are provided, and have to explore, form a hypothesis, act on it, and adapt. It is the difference between a written exam and being handed an unfamiliar machine and told to make it work. That is a much better proxy for the thing people mean when they say an agent can handle a situation it has never seen.

Now the part that got mangled. ARC Prize maintains two very different kinds of result. Semi-private testing, which the organization runs on held-out tasks under controlled conditions, applies only to the first two benchmark generations. Everything else, including all current ARC-AGI-3 entries, sits on the community leaderboard, on the public task set, submitted by whoever ran it. ARC Prize states in its verified testing policy that it does not independently verify self-reported submissions except in extraordinary cases. That is not a knock on NOOA. It is a label the organization prints on its own scoreboard, and it is exactly the label that fell off the claim in transit.

An analogy: a runner posting a time from a training track with their own stopwatch is not the same as a time from a sanctioned meet with officials. Both may be honest. Only one is a record.

The AVO mix-up is the other half. arXiv 2603.24517 does resolve, and it is a real and reasonably interesting paper called "AVO: Agentic Variation Operators for Autonomous Evolutionary Search." Its idea is to use autonomous coding agents as the mutation step inside an evolutionary search, proposing, repairing, critiquing, and verifying code edits, applied to GPU attention kernels on Blackwell hardware. Two NVIDIA-adjacent results, both agentic, both floating around the same week, one benchmark number attached to the wrong paper. That is how most benchmark misinformation actually happens. Not fabrication, just two true things fused into one false thing.

Why this matters beyond the correction: ARC-AGI results move markets of attention. This site has covered a 150-million-parameter model that set an ARC-AGI record for cost rather than score and two API settings that tripled OpenAI's ARC-AGI-3 score without touching the model. Both of those stories exist because the interesting variable in agentic benchmarking is increasingly the harness, the budget, and the reporting conditions, not the raw capability. A number without its conditions attached is close to meaningless, and 85.1% on a public set that anyone can study is a different animal from 85.1% on held-out tasks.

The $332 figure deserves a mention on its own. Interactive benchmarks are expensive because the agent has to actually play, repeatedly, and that cost is now a reported column rather than a footnote. Score-per-dollar is quietly becoming the number worth watching, since test-time compute will buy you a lot of accuracy if you are willing to pay for it.

The honest caveat: none of this means NOOA is not a genuinely strong result. Eighty-five percent on interactive environments with no instructions is impressive by any reading, and self-reported does not mean wrong. It means unaudited. The correct sentence is "an NVIDIA-labelled agent reports 85.1% on the ARC-AGI-3 public demo set," and that sentence is both accurate and still interesting, which is usually the case with the accurate version.

Primary source, verified: read the paper →

Key questions

What makes ARC-AGI-3 different from earlier ARC benchmarks?

It is interactive rather than static. ARC Prize places an agent in novel environments with no instructions provided, so the agent has to work out the rules by acting and observing consequences instead of reading a task description.

Why does public-set versus private-set matter so much here?

A public set can be studied, tuned against, and retried, which makes a high score much weaker evidence of genuine generalization. ARC Prize's policy states that outside its semi-private testing for the first two benchmark generations, results are on the public set and self-reported.

What is AVO actually about?

AVO stands for Agentic Variation Operators, a method in which autonomous coding agents propose, repair, critique, and verify code edits inside an evolutionary search loop. The paper applies it to optimizing attention kernels for NVIDIA's Blackwell GPUs, not to solving puzzle environments.
Cite this

APA

Ground Truth. (2026, August 21). The ARC-AGI-3 record going around is the wrong number and the wrong system. Ground Truth. https://groundtruth.day/news/the-arc-agi-3-record-going-around-is-the-wrong-number-and-the-wrong-system.html

BibTeX

@misc{groundtruth:the-arc-agi-3-record-going-around-is-the-wrong-number-and-the-wrong-system,
  title  = {The ARC-AGI-3 record going around is the wrong number and the wrong system},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/the-arc-agi-3-record-going-around-is-the-wrong-number-and-the-wrong-system.html}
}

Topics: benchmarks · agents · arc-agi · evaluation · nvidia

Related on Ground Truth