Ground Truth.
AI, checked against the source.

Learn · Intermediate

Counter-based random numbers: why the same seed gives different weights on different GPUs

A counter-based random number generator produces each random number as a pure function of a key and a counter, with no hidden state carried from one draw to the next. That design, exemplified by the Philox generator that Salmon, Moraes, Dror and Shaw described in their 2011 paper "Parallel random numbers: as easy as 1, 2, 3," is what lets thousands of GPU threads draw random numbers at once. It also explains a surprising recent finding: because the way work is split across threads depends on the chip, the same seed can deal the same numbers into different positions on different hardware.

The problem it solves

The classic way to make random numbers is a stateful generator: keep a number, scramble it to get the next one, and repeat. That is inherently sequential, because number 1,000 requires numbers 1 through 999. On a GPU with tens of thousands of threads, a single shared sequence becomes a bottleneck. You could give each thread its own generator, but then you must ensure the streams do not overlap, and results depend on how many threads you used.

Counter-based generators sidestep this. Think of a very large, shuffled book of random numbers where the seed picks the book and the counter picks the page. Any thread can open the book at any page without reading the pages before it. Philox does this with a few rounds of a mixing function, much like a lightweight block cipher, applied to the counter with the seed as the key. NVIDIA's cuRAND library ships Philox, and PyTorch uses it for random operations on CUDA, as its reproducibility notes discuss.

Where the hardware sneaks in

A seed alone does not say which thread draws which counter. When PyTorch fills a large weight matrix with random numbers, it launches a grid of threads and each thread generates numbers for a set of positions. The number of threads depends on the GPU: the work is split in proportion to the number of streaming multiprocessors (SMs), the compute units on the chip. A report published on November 12, 2026 by Will DePue of Unemployment Labs gives the counts: 132 SMs on Hopper chips, 148 on B200 and B300, 152 on GB200 and GB300, with 2,048 threads per SM on large jobs.

The consequence is that the same seed with a different SM count produces the same numbers in different positions. In the report's toy example only 6% of positions line up. To reproduce a given set of initial weights you need both the right seed and the right layout. That is the opposite of what many people assume when they set torch.manual_seed: a fixed seed gives reproducible results on the same setup, but the setup includes the chip.

Initialization leaves a trace

Weight initialization, covered in our lesson on weight initialization, sets a model's starting numbers, and training then changes them. But some weights are never touched. Embedding rows for tokens that never appear in training keep their initial values, often exactly. Trained weights keep a faint correlation with their starts. The report uses both. For DeepSeek-V4-Flash it finds a correlation of 0.066, about 57 standard deviations, under the Blackwell layout, and none under the Hopper layout. Whether a given reading proves where a model was trained is a separate question, and the report itself says the fingerprint supports but does not prove it. See our news item on the report for the findings and caveats.

A tool, with limits

The same property that makes this a forensic tool makes it fragile. Anyone who changes the seed, initializes on the CPU, or uses a framework whose generator has no chip layout (the report notes PaddlePaddle's does not) leaves no such fingerprint. Initializing on one system and training on another also breaks the link between the fingerprint and the training hardware. And the technique only works for weights initialized with a known generator and a seed the analyst can find.

What to remember

Key papers
Parallel random numbers: as easy as 1, 2, 3 (Salmon, Moraes, Dror and Shaw, 2011)
cuRAND documentation (NVIDIA)
PyTorch notes on reproducibility

Key questions

What is a counter-based random number generator?

It is a generator with no running internal state: each output is a pure function of a key (the seed) and a counter (the position), so any number in the stream can be computed independently. Philox, introduced by Salmon and colleagues in 2011, is the best-known example and is used by PyTorch on GPUs.

Why does the same seed give different numbers on different hardware?

The generator is the same, but how the work is split across threads is not. If the thread count changes with the chip, the same stream of numbers is dealt into different positions of the weight matrix.

Is a fixed seed enough for reproducibility?

Only if the hardware layout, library versions and the order of operations also match. A seed fixes the stream of numbers, not where each number goes or what rounding the hardware does.
Cite this

APA

Ground Truth. (2026, November 13). Counter-based random numbers: why the same seed gives different weights on different GPUs. Ground Truth. https://groundtruth.day/learn/counter-based-random-numbers-and-why-seeds-depend-on-hardware.html

BibTeX

@misc{groundtruth:counter-based-random-numbers-and-why-seeds-depend-on-hardware,
  title  = {Counter-based random numbers: why the same seed gives different weights on different GPUs},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/counter-based-random-numbers-and-why-seeds-depend-on-hardware.html}
}

Topics: random-numbers · reproducibility · gpu · initialization · pytorch · forensics

Related on Ground Truth