Learn · Intermediate

Mixed-precision training: why models are trained in half-broken numbers on purpose

Mixed-precision training is the practice of training a neural network using deliberately imprecise numbers -- 16 bits instead of 32, and increasingly 8 -- for the bulk of the arithmetic, while keeping full precision at the few points where low precision would break the math. It roughly doubles training speed and halves memory, and it is now universal: essentially every large model you have heard of was trained this way. The technique was established by NVIDIA and Baidu researchers in the 2017 paper Mixed Precision Training, and the details of where precision is kept are what make it work.

What precision actually means

A number in a computer is stored as a sign, an exponent, and a fraction. The exponent controls range -- how large or small the number can get. The fraction controls precision -- how finely you can distinguish nearby values.

The old default, fp32, uses 32 bits: 8 for exponent, 23 for fraction. Enormous range, fine precision, and twice the memory and memory traffic of the alternative.

fp16 uses 16 bits with 5 for exponent and 10 for fraction. Half the memory, and on modern accelerators the matrix-multiply hardware runs it several times faster. But the range shrinks dramatically -- the smallest representable normal value is around 6 times 10 to the negative fifth. Gradients in a deep network routinely land below that, and when they do they become exactly zero. The signal does not degrade; it vanishes.

bfloat16 makes the opposite trade: 8 exponent bits and 7 fraction bits. It has the same range as fp32 with much coarser precision, and Google's work described in A Study of BFLOAT16 for Deep Learning Training established that neural networks tolerate coarse precision far better than they tolerate values silently becoming zero. That insight is why bfloat16 became the default format for large-model training.

The two tricks that make it work

Loss scaling. For fp16, the fix for vanishing gradients is embarrassingly simple: multiply the loss by a large constant before computing gradients, so everything scales up out of the danger zone, then divide the gradients back down before applying them. Modern frameworks adjust the constant automatically, raising it when things are stable and backing off when a gradient overflows to infinity. bfloat16's wide range means it usually needs none of this.

A master copy of the weights. This is the more conceptually interesting one. Weight updates are frequently thousands of times smaller than the weights they modify. Add 0.0001 to 1.0 in 16-bit arithmetic and the result rounds straight back to 1.0 -- the update is silently discarded. So training keeps a full 32-bit master copy of every weight, applies updates to that, and casts down to 16 bits for each step's computation. The tiny increments accumulate faithfully in the master copy even though no individual one would survive in low precision.

The analogy: measuring with a ruler marked only in centimeters while keeping a precise running total in a notebook. Each individual measurement is coarse, but the ledger does not lose the fractions.

Certain operations also stay in higher precision by convention -- softmax, layer normalization, and the accumulation inside a matrix multiply -- because they involve summing many values or exponentiating, where small errors compound.

Going to eight bits

The frontier has moved to fp8, standardized in FP8 Formats for Deep Learning in two variants: one with more exponent bits for gradients, one with more fraction bits for weights and activations. With only 8 bits total, a single global scale factor is hopeless, so fp8 training uses per-tensor scaling factors tracked and updated as training proceeds -- essentially loss scaling generalized to every tensor. Several recent large open-weight models ship their checkpoints in fp8, which is why anyone fine-tuning them often has to convert to bfloat16 first.

The same pressure applies to the optimizer's own bookkeeping, which is often larger than the model. 8-bit Optimizers via Block-wise Quantization compresses that state by quantizing it in small blocks, each with its own scale, and Adafactor attacks it differently by storing row and column summaries instead of a full per-parameter table.

The failure mode nobody talks about

Low precision does not usually announce itself. It shows up as things quietly not happening.

A recent optimizer paper illustrates this perfectly. Its author found that weight decay -- the regularization technique that gently shrinks weights toward zero to prevent overfitting -- was effectively inert in every configuration tested, because the decay amount was small enough to round away under bfloat16. Every baseline in the paper nominally used weight decay. None of them actually got any.

That is the characteristic hazard. Nothing crashed, no warning fired, and every number in the results table looked reasonable. A hyperparameter simply had no effect, and it took someone specifically checking to notice.

Why this matters

Mixed precision is one of a small handful of changes that made models at today's scale economically possible -- alongside distributed parallelism and better optimizers. It is also a useful mental model for the whole field's relationship with numbers: neural networks are remarkably tolerant of noise in the individual values and remarkably intolerant of values disappearing entirely. Nearly every precision technique that works is an application of that one asymmetry.

It is closely related to but distinct from quantization, which shrinks an already-trained model for cheaper inference. Mixed precision is about the training run itself, where gradients still have to flow.

Ground Truth. (2026, July 23). Mixed-precision training: why models are trained in half-broken numbers on purpose. Ground Truth. https://groundtruth.day/learn/mixed-precision-training.html

@misc{groundtruth:mixed-precision-training,
  title  = {Mixed-precision training: why models are trained in half-broken numbers on purpose},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/mixed-precision-training.html}
}

Topics: training · efficiency · precision · hardware · fundamentals