News · 2026-09-07

GPT-6 Astra’s conflicting benchmark positions show why the harness now matters as much as the model

GPT-6 Astra’s public benchmark standing changes markedly with the evaluation surface, including ARC-AGI-3 scores of 62.7% and 99.9% under different harnesses. The news is not that a leaderboard has chosen a universal winner; it is that the practical system around a model is now inseparable from the result being measured.

Key facts

• ARC Prize reported the two Astra Semi-Private results on September 2.
• The Standard harness score was 62.7%; the Provider Adapter harness score was 99.9%.
• ARC Prize explicitly says the results are not proof of AGI.
• WebDev Arena lists GPT-6 Astra Max first, while LiveBench lists Astra Max Effort third on its latest release table.

A benchmark result is often described as though a model walked into an exam alone. In practice, an agentic evaluation is closer to measuring a pit crew plus a car. The harness decides what instructions are used, whether tools are available, how errors are retried, whether the provider’s own adapter is involved, what context is preserved and how results are scored. Change those rules and a different system is on the track.

ARC Prize’s numbers make the effect unusually vivid: 62.7% in one arrangement and 99.9% in another. Neither number is fraudulent simply because they differ; each describes a different experimental setup. But any headline that omits the harness risks treating an integration result as an intrinsic property of the weights. That is why the source’s own sentence matters: “This is not proof of AGI.”

Other tables are measuring other things. Artificial Analysis shows a tight frontier cluster, while its release comparison can show ties that its broader model page presents differently. MazeBench reports that no model exceeded 1% without Python, with frontier systems around 10% when the tool is available. ClockBench puts the leading model at 66.7% versus a 90.7% human baseline. Signal65 PINNACLE runs 280 code-verified enterprise jobs. These are not interchangeable intelligence meters; they are probes of different capabilities and constraints.

The research literature adds a reason to be cautious about black-box leaderboards. Clean Engineering, Unstable Measurement reports 52,988 audited request attempts and only 0.400 Spearman correlation for same-window repeat rankings, against a 0.90 target. Its authors argue that shared endpoints can be unstable enough that the measurement instrument itself needs monitoring. A companion paper, Conformity Breaks Conformal Prediction, finds coverage dropping from 90% to 74% when peers are unanimously wrong.

The strongest counterargument is that readers still need a simple comparison, and standard leaderboards offer an accessible starting point. That is true. The alternative is not paralysis; it is better labels. A useful result should identify the exact model version, date, provider settings, tools, harness, scoring rule and whether the task resembles the intended deployment.

Why it matters: model shopping, safety claims and policy decisions now depend on systems rather than standalone models. The right question is not “which model won?” but “which configured system performed on which task, under what conditions, and how stable was the measurement?”

Primary source, verified: read the paper →

Ground Truth. (2026, September 7). GPT-6 Astra’s conflicting benchmark positions show why the harness now matters as much as the model. Ground Truth. https://groundtruth.day/news/benchmarks-put-gpt-6-astra-in-different-places.html

@misc{groundtruth:benchmarks-put-gpt-6-astra-in-different-places,
  title  = {GPT-6 Astra’s conflicting benchmark positions show why the harness now matters as much as the model},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/news/benchmarks-put-gpt-6-astra-in-different-places.html}
}

Topics: benchmarks · evaluation · openai · agents · reliability