News · 2026-10-26

A new test checks whether an AI model’s probabilities fit together

Peter Potash's Latent Coherence study finds that frontier language models can give probability estimates that disagree with the estimates implied by their own component judgments. Larger disagreements tended to accompany larger errors in every tested model. The check can reveal a reason for caution, but it does not reliably tell a user which of the conflicting estimates to trust.

Key facts

• The study evaluated seven models on 1,590 shared prompts per model.
• Its first arXiv version was submitted October 22, 2026.
• It tests a probability identity using separate calls, with a repeated-wording subset to distinguish instability from persistent disagreement.
• The primary source is Latent Coherence; full text and code and data are available.

The idea begins with a rule that is easy to illustrate. Suppose, in a hypothetical inspection problem, a shipment has a 40% chance of coming from one factory and a 60% chance of coming from another. If the defect probabilities are 70% and 20% respectively, the overall defect probability is 40%. The pieces have to combine that way if those factories cover all possibilities and the statements refer to the same evidence.

A model could nevertheless answer 70% when asked directly about the shipment. That would leave a disagreement between two routes to the same quantity. Potash's paper asks whether models obey what its title calls the “Law of Total Probability,” using richer cases involving hidden causes and evidence.

The method does not simply ask the model to reconsider its first answer. It collects the direct estimate, probabilities of the hidden cause's possible states, and conditional estimates through isolated calls. The answers cannot be harmonized merely because the model was shown its earlier numbers. A subset of items is also presented in three wordings to examine how much disagreement survives changes in phrasing.

That separates two different problems. One is ordinary response instability: asking a similar question produces a different number. The other is a stable mismatch between the direct answer and the combined component estimates. The second problem can persist even if repeatedly asking the same question gives a reassuringly consistent response.

The analogy is a budget assembled by separate departments. A company can have stable departmental forecasts and a stable company-wide forecast that still do not add up. Repeating the forecasts will not repair the mismatch. Someone has to inspect the assumptions and relationships between the numbers.

The paper compares this disagreement with known answers rather than assuming that mathematical consistency equals truth. Within each model, larger residuals tended to accompany larger errors. However, the paper finds that sometimes the direct estimate is closer and sometimes the decomposed estimate is closer. Breaking a probability into components did not reliably improve accuracy.

This is why coherence should remain distinct from calibration. Calibration concerns whether events assigned a probability occur at the corresponding frequency. Coherence concerns whether related beliefs fit together mathematically. A system can be internally consistent and still consistently wrong about the world. Conversely, a fairly accurate answer can coexist with incompatible supporting estimates.

The units in the reported results deserve particular care. The two most coherent models had mean absolute residuals of 0.08 and 0.12 in log-odds. The abstract describes their performance in percentage-point language, but log-odds are not percentage points. The same log-odds movement corresponds to different probability changes near the middle and the ends of the probability range. Treating one as the other would replace a qualified result with an unjustified universal number.

The repository makes the evaluation inspectable, with an MIT license for code and a separate Creative Commons license for data. That is useful for reproduction. It does not constitute independent replication, and the dossier contains no independent expert review establishing that the result generalizes to every decision-support setting.

There are also benchmark boundaries. The study uses seven models at one point in time, with exact answers known for the test cases. Some qualitative examples were fictionalized with language-model assistance. Simple normalization and causal adjustment checks mostly passed, while harder qualitative items produced larger residuals. The result therefore does not support the blanket claim that models cannot perform probability arithmetic.

For builders composing several model judgments, the study offers a concrete diagnostic: identify relationships the judgments ought to satisfy and test them separately from overall task accuracy. Bayesian updating explains another part of that probability toolkit. The caveat is that finding a broken relationship does not automatically repair it. Coherence is a warning signal and a property worth measuring, not a substitute for checking what is true.

Primary source, verified: read the paper → (arXiv 2610.31833)

Ground Truth. (2026, October 26). A new test checks whether an AI model’s probabilities fit together. Ground Truth. https://groundtruth.day/news/latent-coherence-tests-whether-ai-probabilities-fit-together.html

@misc{groundtruth:latent-coherence-tests-whether-ai-probabilities-fit-together,
  title  = {A new test checks whether an AI model’s probabilities fit together},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/latent-coherence-tests-whether-ai-probabilities-fit-together.html}
}

Topics: research · evaluation · uncertainty · probability · reliability