Learn · Intermediate

Post-Training Quantization: Calibration, GPTQ and Why Rotations Help

Post-training quantization (PTQ) shrinks an already-trained model to low-precision numbers, usually 4 bits per weight, without retraining it, by using a small calibration set of example inputs to decide how to round. The state-of-the-art recipes combine three ideas covered here: scaling, error-compensating rounding as in GPTQ, and rotations that tame outliers. A new open tool, Huawei's MACaQ, stacks all three. For the basics, start with our lesson on quantization.

The problem: rounding hurts, unevenly

Quantizing means mapping each real-valued weight onto one of a few levels. With 4 bits there are 16 levels, spread between the layer's smallest and largest values. Two things go wrong. First, plain round-to-nearest treats every weight independently, but errors in different weights add up in the layer's output, so a set of individually small errors can combine into a large one. Second, a handful of outlier values stretch the range, so the 16 levels are spaced far apart and most ordinary weights are rounded coarsely. Large language models are known to have such outliers, especially in activations.

Calibration: letting data pick the rounding

PTQ methods feed a few hundred example inputs, the calibration set, through the model and record what each layer sees. The layer is then treated as a small problem: choose quantized weights so the layer's output on those inputs stays as close as possible to the original. Doing this one layer at a time keeps memory small, which is why a 122-billion-parameter model can be quantized on a single modest machine. The catch is that calibration data biases the result: a calibration set unlike real usage can leave the quantized model worse on tasks it did not see.

GPTQ: round, then repair

Frantar and colleagues' 2022 GPTQ paper made a layer-wise approach practical for billion-parameter models. For each weight row, it quantizes weights one at a time. After rounding a weight, it knows the error it introduced, and it spreads a correction across the remaining unrounded weights so the layer's output is preserved. How much to adjust each one comes from the Hessian, the second-order curvature of the layer's squared output error, which is computed from the calibration inputs. It is like tiling a floor: when one tile is cut slightly short, you adjust the gap sizes for the tiles still to be laid so the overall edge stays straight. Later methods refine this idea; Huawei's "Flip-Refinement" is described as a Hessian-aware coordinate descent that nudges individual 4-bit codes away from round-to-nearest.

Rotations: making outliers disappear

The outlier problem needs a different fix. A Hadamard matrix is a square grid of +1 and -1 entries whose rows are mutually perpendicular; scaled properly it is a rotation. Multiplying a vector containing one huge value by it spreads that value's energy across all entries, so the result has no extreme spikes and quantizes well. The trick that makes this free in practice is that a rotation can be undone. If you rotate the activations going into a layer and apply the inverse rotation to the layer's weights, the product is unchanged mathematically, so the network computes the same function, but both sides now have smoother numbers. QuaRot (Ashkboos and colleagues, 2024) showed this lets both weights and activations go to 4 bits with outlier-free inputs. The rotations for weights are folded in ahead of time; some online ones, kept small and fast, remain.

Measuring the damage honestly

How do you know a quantized model is good? Perplexity and benchmark accuracy are common, but a cleaner direct measure is KL divergence between the quantized model's next-word probabilities and the original's on the same text (see forward and reverse KL). Lower means closer to the original. It is a proxy: a model can drift in probabilities without losing task accuracy, and the reverse. MACaQ's authors, for instance, report KL numbers against public GGUF files of the same size.

Practical guidance

Compare files at equal size, not equal bit count, since formats such as GGUF mix bit widths (model file formats). Check which baseline a claimed improvement beats: a gain over naive affine quantization is much easier than a gain over mature K-quants. And remember the disk size saved is not the only benefit: inference is memory-bound, so fewer bytes per weight also means faster generation.

The takeaway

PTQ works because layers can be treated as small, solvable problems. Calibration data tells the rounding what matters, GPTQ-style correction keeps layer outputs honest, and rotations remove the outliers that waste precision. Each trades a bit of upfront work for a model that fits on cheaper hardware.

Ground Truth. (2026, November 12). Post-Training Quantization: Calibration, GPTQ and Why Rotations Help. Ground Truth. https://groundtruth.day/learn/post-training-quantization-calibration-gptq-and-rotations.html

@misc{groundtruth:post-training-quantization-calibration-gptq-and-rotations,
  title  = {Post-Training Quantization: Calibration, GPTQ and Why Rotations Help},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/post-training-quantization-calibration-gptq-and-rotations.html}
}

Topics: quantization · gptq · calibration · hadamard · inference · llm-compression