News · 2026-10-28

Artificial Analysis changes its flagship index to emphasize held-out work

Artificial Analysis has revised its Intelligence Index so that 60 percent of the composite now depends on private tests or held-out answers, while adding more demanding coding and scientific workflows. The October 27 revision puts Claude Opus 5.5 first overall, but its larger significance is a changed definition of the work being measured.

Key facts

• Version 5 contains nine evaluations, with four categories receiving 25 percent weight each.
• Private test sets or held-out answers account for 60 percent of the index.
• Opus 5.5 scores 58, just ahead of GPT-6 Astra at 57 in the publisher’s composite.
• The primary sources are the v5 announcement and methodology.

A leaderboard can move because the models improve, because the exam changes, or both. Version 5 is a clear example of the second effect. It reorganizes the composite around coding, science, general tasks, and the category Artificial Analysis calls “Knowledge Work.” Each category receives an equal share. That makes the score easier to interpret as a set of chosen priorities, rather than a natural unit of intelligence.

Imagine comparing vehicles with a score combining city driving, hauling, off-road travel, and running costs. Changing the balance changes which vehicle wins, even if none of the vehicles changes. An AI index works similarly. Its value comes from making the tasks and weights explicit enough that readers can decide whether those priorities resemble their own work.

Two additions make the new direction concrete. FrontierCode 2.0 asks agents to solve private real-world coding tasks, with the standard of whether a repository maintainer would merge the changes. Terminal-Bench-Science contains 70 terminal-based research tasks. These move beyond producing an attractive answer toward completing a workflow with constraints and checks. They still evaluate the supplied setup, including the surrounding agent system, rather than an abstract model in isolation.

The revision also upgrades several existing components. It replaces the earlier broad exam with HLE-Diamond, updates the knowledge test AA-Omniscience, and uses a private hard subset of the physics test CritPt. Other components remain, while some older tests leave the composite. GDP.pdf’s share drops from 10 percent to 5 percent. That detail matters because a headline ranking is partly an editorial decision about what deserves weight.

Private tests and held-out answers address a real problem. Public questions can enter training material or become targets for repeated tuning, making familiarity look like general capability. Ground Truth’s benchmark-contamination lesson explains that failure mode. Holding information back can reduce exposure, but the word private is not a complete quality guarantee. A test can be private and still narrow, unstable, or poorly matched to a particular job.

The new ranking places Opus 5.5 at 58, GPT-6 Astra at 57, and GPT-6.1 Sol at 55. Those values are composite scores, not success percentages or odds that a response is correct. A one-point difference should not become a universal recommendation. The task-level result is more useful: Astra leads the scientific terminal evaluation and the hard physics subset, while other systems lead other components.

The release also highlights an openness distinction. Kimi K3.1 Pro is listed at 46, but its weights were not yet released. Calling it downloadable would turn a prospective status into an available product. The strongest released-weight entries listed score lower. The correct reading is that the gap changes under this particular suite; it does not establish that every local model is unsuitable for every useful task.

Cognition’s FrontierCode methodology provides context for why tougher repository standards can reorder models. A small Hacker News discussion questions how sharply the new version differs from the old one. That is a legitimate reproducibility and comparability concern, but not an independent audit of the whole index. The dossier found no verified independent replication or expert consensus for v5.

A separate circulating claim needs to stay separate. The headline about AI completing 37 percent of real engineering tasks concerns Xplore’s RWE-bench, not Artificial Analysis. The dossier’s primary-page check established its engineering-task design but did not expose the numerical leaderboard details needed to confirm that figure. It also found no published human pass-rate comparison supporting claims that ordinary people would pass none.

The practical use of v5 is to find the evaluation closest to a real task, inspect what counts as success, and then test a short list in the intended workflow. Ground Truth’s benchmarking lesson gives that framework. A better exam can reveal important differences. It cannot decide which assistant is best for every reader.

Primary source, verified: read the paper →

Ground Truth. (2026, October 28). Artificial Analysis changes its flagship index to emphasize held-out work. Ground Truth. https://groundtruth.day/news/artificial-analysis-index-v5-held-out-workflows.html

@misc{groundtruth:artificial-analysis-index-v5-held-out-workflows,
  title  = {Artificial Analysis changes its flagship index to emphasize held-out work},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/artificial-analysis-index-v5-held-out-workflows.html}
}

Topics: benchmarks · evaluation · artificial-analysis · coding-agents · ai-for-science