Learn · Intermediate

Why temperature zero is not deterministic

Setting temperature to zero does not make a language model deterministic. It makes the selection rule deterministic, which means the model always takes its highest-scoring next token instead of sampling. But the scores themselves move slightly between runs, and when two candidate tokens are nearly tied, a tiny shift flips which one wins. That single flip changes the next token, and the next, and a hundred tokens later you have a visibly different answer. This is why the same prompt sent twice to the same hosted model at temperature zero can come back different, and why evaluation results wobble on models that were never retrained.

The root cause is that floating-point addition is not associative. In exact mathematics, adding a group of numbers gives the same total no matter how you group them. In floating point, each addition rounds, so grouping changes the rounding and therefore the total, usually in the last few bits. That sounds negligible, and for one addition it is. A transformer forward pass performs an enormous number of these sums, and the differences compound through layers.

Why would the grouping ever change? Because GPUs get their speed by splitting a sum across thousands of parallel workers and combining the partial results as they finish. The combination order depends on how the work was divided, which depends on the batch: how many requests are being served together, how long each one is, how the scheduler packed them. Your request being processed alongside seven others produces a slightly different set of partial sums than the same request processed alongside two. This is the crux of the argument in Thinking Machines' Defeating Nondeterminism in LLM Inference, which identifies batch-dependent kernel behavior, rather than raw parallel-reduction randomness, as the main practical culprit in real serving stacks. The proposed fix is batch-invariant kernels: implementations that compute the same result regardless of how many other requests share the batch.

Several more sources stack on top. Attention implementations like FlashAttention tile the computation to keep data in fast memory, and the tiling depends on sequence length, so different-length inputs take different arithmetic paths. Libraries auto-select among several kernel implementations based on shapes and available memory, which means a different kernel can be chosen on an otherwise identical run. Mixture-of-experts models route each token to a subset of experts, and when routing is capacity-limited per batch, which experts a token gets depends on what else is in the batch, so a token can be processed by different weights entirely. And speculative decoding accepts or rejects draft tokens based on comparisons that are themselves subject to the same floating-point drift.

The practical consequences fall into three buckets. First, debugging: a bug you cannot reproduce may not be intermittent in your code at all. Second, caching and testing: exact-match assertions on model output are fragile, and test suites built on them fail randomly. Third, and most damaging, evaluation. If you score a model by generating one answer per problem and marking it right or wrong, some borderline problems flip between runs. The 2026 audit Phantom Gains showed that a frozen model, unchanged in every way, appeared to both learn and forget under exactly this effect, which means any metric that counts newly solved problems reports a nonzero result on a model that did nothing. That is the direct link between this engineering detail and the research-methods problem of null baselines and multiple comparisons.

If you need reproducibility, it is achievable but not free. Fix the batch size, ideally to one. Pin the library versions and disable auto-selection of kernels. Use batch-invariant kernel implementations where your stack offers them. Set every random seed, including the ones in your sampling and data-loading code. Run on identical hardware, since different GPU generations have different instruction sets and reduction widths. All of this costs throughput, which is precisely why hosted inference providers do not do it by default: serving is an economics problem, and batching many requests together is where the economics come from. Determinism and utilization pull against each other.

The better habit, for anyone evaluating models, is to stop asking for reproducibility and start measuring the noise. Generate several samples per problem instead of one. Report variance across runs alongside the mean. Compare systems on the same problems, paired, so that problem difficulty cancels out. And when a result depends on a handful of problems moving, check whether an unchanged model moves that many on its own. Understanding that temperature zero is a rule about selection and not a promise about output is the first step; the rest follows from treating model outputs as measurements with error bars, which is what they have always been. This also reframes how a model picks its next word: the dice roll is only one of the random-looking things happening, and turning it off leaves the others running.

Ground Truth. (2026, August 22). Why temperature zero is not deterministic. Ground Truth. https://groundtruth.day/learn/why-temperature-zero-is-not-deterministic.html

@misc{groundtruth:why-temperature-zero-is-not-deterministic,
  title  = {Why temperature zero is not deterministic},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/why-temperature-zero-is-not-deterministic.html}
}

Topics: inference · reproducibility · gpu · evaluation · engineering