News · 2026-11-09

Independent monitor finds no qualifying Opus 5.5 change on its fixed test

LiveNerf’s independent monitor found no qualifying change in Claude Opus 5.5 accuracy across its first three completed comparison windows through 8 November. The result limits claims of a broad, large change on its fixed academic question set, but it does not settle separate user reports of declining coding-agent performance.

Key facts

• LiveNerf tests a fixed panel of 78 academic questions each day.
• Its day-46 update reported “no change” under its preregistered rule in all three completed windows.
• A separate user report describes a rise in correction-tagged answers around 19 October.
• Neither the personal logs nor a server-side change were independently verified in the dossier.

LiveNerf compares each question with its own baseline from the first ten days. It uses exact-match grading and a separate control model, with a fixed client and a tool-free environment. Its design record sets a demanding rule: a qualifying change must clear a 99% interval threshold in two consecutive windows, exceed three percentage points, and not coincide with a same-direction control move. The project estimates that its realized detection limit is about 6.6 percentage points per ten-day window.

That is like putting a thermometer in one room and checking it at the same time every day. Stable readings make a major change in that room less likely; they do not measure the temperature in every other room. Here the unmeasured rooms include multi-file coding, long sessions, tool use and task-specific instruction following. The project's validation also says it has not shown that its longer window would reliably detect a same-family model substitution of the tested magnitude.

A Claude Code user separately reported that the share of answers followed by a correction or self-retraction rose from roughly one in twenty to roughly one in four across the relevant periods. That is more concrete than a vague impression, but a correction is an imperfect measure of an error, the underlying transcripts were not independently available, and session material and effort settings changed around the investigation. A second report described a similar experience on Windows; it is another uncontrolled observation, not a replication of the same measurement.

Anthropic’s model versioning documentation distinguishes model weights from serving infrastructure, which can change independently. That general caveat does not establish a change in this case, particularly because LiveNerf uses a subscription and Claude Code path. Its fixed client and label cannot expose every backend decision.

The strongest counterargument to the null result is therefore not that the monitor is useless; it is that it asks a narrower question than affected users do. Conversely, personal complaints cannot prove a provider changed weights or settings. The next useful evidence would be a preregistered, repeatable coding workflow with archived inputs and outcomes, alongside disclosure of any relevant provider-side changes. Paired comparisons explain why reusing the same cases helps, and why the choice of cases still matters.

A fixed panel also makes tradeoffs in what it deliberately leaves out. LiveNerf removes tools, user hooks, memory and project instructions to reduce variation in the measurement. That helps isolate the behavior of its particular question-answering setup, but it removes many features that make an everyday coding assistant useful. When a user says a coding agent is worse, they may mean it edits the wrong file, misunderstands a repository, follows an instruction inconsistently, or requires more supervision. None of those outcomes is directly represented by an exact-match academic answer.

The opposite danger is treating a change in a personal workflow as proof that the model itself changed. Projects evolve, prompts vary, and a user’s expectations and selection of tasks can change. The reporter documented several local checks, which makes the claim worth following, but could not inspect the provider’s hidden serving path. Those unobserved details constrain causal conclusions in either direction.

A useful follow-up would preserve a collection of representative coding tasks and their input repositories, specify success criteria before running them, and repeatedly evaluate the same tasks with matching settings. Independent reviewers could inspect changes and tests rather than simply counting the agent’s own declarations of success. It would still be necessary to say which tasks were sampled and what size of difference the study could detect.

The debate illustrates why monitoring a service is more than watching a model name. A stable label is useful for users, but experience can also reflect the surrounding client, tools, instructions, or service. The available record identifies no verified change to any one of those elements in the disputed period. The appropriate conclusion remains a bounded benchmark finding alongside unresolved reports.

Primary source, verified: read the paper →

Ground Truth. (2026, November 9). Independent monitor finds no qualifying Opus 5.5 change on its fixed test. Ground Truth. https://groundtruth.day/news/livenerf-opus-55-drift-debate.html

@misc{groundtruth:livenerf-opus-55-drift-debate,
  title  = {Independent monitor finds no qualifying Opus 5.5 change on its fixed test},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/news/livenerf-opus-55-drift-debate.html}
}

Topics: evaluation · anthropic · model-drift