News · 2026-09-03

OpenAI shipped GPT-6 Astra, and its headline benchmark score has two different answers

OpenAI began rolling out GPT-6 Astra on September 3, 2026, pricing it at $10 per million input tokens and $50 per million output tokens -- materially above its previous flagship. The launch's marquee claim, a near-perfect score on the ARC-AGI-3 reasoning benchmark, turns out to depend entirely on which test harness ran it: ARC Prize's own results page shows 62.71% under one harness and 99.95% under another, for the same model on the same test.

Key facts

• GPT-6 Astra's best observed ARC-AGI-3 Semi-Private result was 62.71% under the Standard harness (cost: $26,098) and 99.95% under the Provider Adapter harness (cost: $18,817), per ARC Prize's results page.
• Pricing: $10 per million input tokens, $1 cached input, $12.50 cache writes, $50 output, per the OpenAI API model page.
• Announced and began rolling out September 3, 2026 -- to a limited set of organizations first, then Plus, Pro, Business, Enterprise, the API and AWS over the following days.
• Primary source: OpenAI, "GPT-6 Astra: A new generation of intelligence".

The gap is the story. A test harness is the software wrapper that sits between a model and a task -- it decides how the environment is described, how many attempts are allowed, how tool calls are formatted, how errors are retried. It is plumbing, and for years nobody reported it because nobody thought it mattered much. On ARC-AGI-3, a benchmark of interactive puzzle environments designed to resist memorisation, the plumbing moved the result by more than 37 percentage points.

Think of it like timing a runner. Same athlete, same distance, but one clock starts when the gun fires and the other starts when they cross the first sensor. Both times are honestly measured. Only one of them answers the question you asked. OpenAI's launch page leads with the near-perfect number. ARC Prize published both, and its blog rounds them to 62.7% for about $26,000 and 99.9% for about $19,000 -- note that the worse score cost more, which is what happens when a model flails against a harness that gives it less structure.

None of this makes Astra unimpressive. ARC Prize reports something genuinely striking alongside the scores: Astra used fewer actions than the median tested human on 96.0% of levels. In ARC Prize's description, the model turns unfamiliar environments into compact symbolic world models and invents its own shorthand for tracking state and planning. That is a claim about efficiency and representation, not just accuracy, and it is harder to game with harness choice.

The model itself is aimed squarely at agentic work. OpenAI's API documentation calls Astra "our most capable model" for hard end-to-end tasks, and the launch post frames it around computer use, browsers, coding and professional workflows -- multi-step jobs where the model asks focused clarifying questions when the answer would change the outcome, and keeps working asynchronously while a tool runs. Developers get five reasoning effort levels: low, medium, high, xhigh and max. There is no off switch; OpenAI's model guidance page states plainly that Astra does not support the none effort setting, which means any benchmark row labelled "None" is an evaluation condition researchers created, not something a user can pick.

Access is narrower than the announcement implies. OpenAI's help documentation says that for ChatGPT Business, Standard seats get limited Astra usage inside their existing Work and Codex allowance, while Premium seats can spend their full existing allowance on it once it appears in the workspace. The "unlimited" language in the marketing applies to Instant chat, not to reasoning, Work or Codex. And with Fast mode billed at 2.5 times standard inside Work and Codex, prompt caching stops being an optimisation and becomes the difference between an agent loop you can afford and one you cannot -- an economics problem our explainer on inference cost and token economics covers in detail.

The reception on Hacker News, where the model thread drew 1,373 points and 1,127 comments, was not the reflexive dismissal these launches usually attract. The sharpest objection was not that the capability is fake. It was that the framing is inflated: that the definition of general intelligence is quietly being lowered to whatever the newest model can do, that the system still cannot learn continuously between sessions, and that a scorecard which moves 37 points on harness choice is not a scorecard. A parallel thread on the Artificial Analysis coding-agent index made the economic version -- token efficiency gains get erased when the price per token triples.

Why it matters beyond one launch: the industry's measurement apparatus is now a bigger source of variance in reported capability than the models themselves. That is not an abstract concern. The same week Astra launched, a research paper called HarnessDev found that the harness -- execution loop, tool policy, context management, state, recovery, verification -- determines outcomes so strongly that it deserves to be evaluated as a system in its own right. When vendors choose the harness that produces the headline, benchmark numbers become a marketing surface. Our explainer on how AI is benchmarked walks through why this failure mode keeps recurring.

The honest caveat: both ARC Prize numbers are real, independently published, and were not hidden. ARC Prize deserves credit for putting the unflattering one on the same page as the flattering one -- most benchmark operators would not. The problem is downstream, in how a two-number result collapses into a one-number headline before it reaches anyone making a decision based on it.

Primary source, verified: read the paper →

Ground Truth. (2026, September 3). OpenAI shipped GPT-6 Astra, and its headline benchmark score has two different answers. Ground Truth. https://groundtruth.day/news/astra-scores-62-percent-and-99-percent-on-the-same-benchmark.html

@misc{groundtruth:astra-scores-62-percent-and-99-percent-on-the-same-benchmark,
  title  = {OpenAI shipped GPT-6 Astra, and its headline benchmark score has two different answers},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/news/astra-scores-62-percent-and-99-percent-on-the-same-benchmark.html}
}

Topics: openai · gpt-6-astra · frontier-models · benchmarks · agents · pricing