News · 2026-10-29

Artificial Analysis resets its index, with 60% on private tests or held-out answers

Artificial Analysis rebuilt its intelligence index on October 27, changing the tests and weights used to compare leading models. The organization says 60% of version five depends on private test sets or held-out answers, strengthening protection against direct contamination while limiting exact outside reproduction. The next day’s Carbon ranking should therefore be read on a newly defined scale.

Key facts

• Version five launched October 27 with two new tests, three upgrades, and three removals.
• Knowledge work, coding, science, and general capability each receive equal broad-area weight.
• Artificial Analysis says 60% of the index uses private test sets or held-out answers.
• Primary source: Artificial Analysis’s v5 methodology announcement.

The headline is a measurement change before it is a model race. An index combines results from different tasks into one number. Deciding which tasks count, how difficult they are, and how much weight each receives defines what the final number rewards. A model can move on that number even when its underlying behavior has not changed, because the measuring instrument has.

Artificial Analysis’s nine component evaluations span a deliberately broad set of work. The updated collection includes repository changes, terminal tasks, professional knowledge work, scientific problems, factual accuracy, and multi-application workflows. The organization labels its rebuilt composite “Intelligence Index v5.” That version label matters whenever a score is compared with an earlier screenshot or a historical claim of leadership.

A school-report analogy is useful. A student’s average changes if the school adds a difficult laboratory exam and increases the weight of mathematics. That new average may be more informative about the school’s goals, but the change does not prove that the student became better or worse overnight. In the same way, index revisions can improve measurement while breaking a simple historical comparison.

The case for hidden tests is substantive. Public benchmark questions can enter training data or become targets for repeated optimization. A model that has encountered the exact problem may appear more capable than it is on unfamiliar work. Artificial Analysis’s benchmarking methodology provides context for its evaluation approach, while our benchmark-contamination lesson explains the failure mode it is trying to reduce.

The cost is incomplete auditability. Public protocols, task descriptions, examples, and weights let outsiders examine the broad construct. They do not let those outsiders inspect every private item, check every grading decision, or rerun the exact hidden test. It is too absolute to say the index cannot be audited at all. It is accurate to say exact independent reproduction is limited.

That tradeoff became immediately visible when Artificial Analysis reported Carbon at number one on October 28. Carbon scored 59 and led four of nine components. The same analysis says it did not lead coding or knowledge work and ranked behind Claude Opus 5.5 on the separate coding-agent index. A single composite makes those different strengths easier to summarize and easier to overlook.

Even a component win has internal tradeoffs. Artificial Analysis reports improved Carbon accuracy on its knowledge evaluation alongside a measured hallucination-rate increase from 4% to 11% compared with Argon. That does not automatically contradict Google’s internal claim of fewer unsupported statements, because the tests and definitions differ. It does show why a broad statement such as better knowledge should not be silently expanded into fewer errors under every condition.

The v5 discussion on Hacker News and the Carbon thread include skepticism about private tests, the timing of the revision, and how much a narrow ranking margin means for ordinary work. Those are legitimate questions about measurement and provenance. The dossier did not establish a named independent analysis proving that the index was designed to favor Google or OpenAI.

Artificial Analysis evaluated Carbon before public availability, but the checked sources do not disclose who provisioned that access. That is an unanswered provenance question. It is not evidence that Google controlled the scoring or that the organization’s result is simply Google’s own table republished. Google’s methodology separately identifies Artificial Analysis as the source of the index result.

The practical response is to read the component results, methodology, and model leaderboard together. Our benchmarking lesson explains why one ranking cannot choose a model for every workflow. Version five may be a useful attempt to measure harder, less contaminated work. Its honest limitation is that a changing, partly private composite offers a particular view of capability, rather than a universally reproducible definition of the best model.

Primary source, verified: read the paper →

Ground Truth. (2026, October 29). Artificial Analysis resets its index, with 60% on private tests or held-out answers. Ground Truth. https://groundtruth.day/news/artificial-analysis-v5-private-tests-ranking-reset.html

@misc{groundtruth:artificial-analysis-v5-private-tests-ranking-reset,
  title  = {Artificial Analysis resets its index, with 60% on private tests or held-out answers},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/artificial-analysis-v5-private-tests-ranking-reset.html}
}

Topics: benchmarks · evaluation · artificial-analysis · transparency · industry