News · 2026-07-11

Nvidia's 'Puzzle' pipeline shrinks a 120B model to 75B and roughly doubles serving throughput

Nvidia released Nemotron-3-Puzzle, a compressed model that takes a 120-billion-parameter parent down to 75 billion while keeping most of its capability -- and, more importantly, roughly doubling how much work a server can do with it. On a single high-end B200 node it delivers about twice the throughput, and at very long context on a single H100 it goes from serving one request at a time to eight. It is a case study in the quiet half of AI progress: not a smarter model, a cheaper one.

Key facts

• Compression: 120B to 75B total parameters (active parameters 12.8B to 9.3B), retaining strong downstream accuracy.
• Throughput: roughly 2x on a single 8xB200 node; 1-million-token concurrency on one H100 rises from 1 request to 8.
• Method: the 'Iterative Puzzle' framework -- joint MoE, active-budget, and Mamba pruning -- plus distillation, RL, NVFP4 quantization, and a multi-token-prediction head.
• Primary source: the Nemotron-3-Puzzle paper (weights on Hugging Face).

The problem here is economics, not intelligence. A frontier-class model that costs a fortune to serve reaches far fewer people than one that runs cheaply, and every large lab now spends serious effort making capable models smaller and faster after training them. The catch is doing it without gutting quality -- naive shrinking makes a model dumber. Nemotron-3-Puzzle is Nvidia's recipe for doing it carefully, and the name 'Puzzle' is literal: it treats compression as assembling the right combination of pieces rather than applying one blunt trick.

Those pieces stack several established compression ideas. Pruning removes parts of the network that contribute least -- and here it prunes in three dimensions at once: which experts a mixture-of-experts layer keeps, how many parameters fire per token, and -- notably -- the model's Mamba layers. That last detail confirms this is a hybrid architecture combining attention with state-space model layers, and Nvidia prunes both. On top of pruning it layers knowledge distillation (training the smaller model to imitate the larger one), reinforcement learning to recover capability, and quantization down to a 4-bit format (NVFP4) that stores each number in far fewer bits. A multi-token-prediction head lets it guess several tokens per step to speed generation. Each technique individually is well known; the contribution is the disciplined pipeline that combines them without the quality collapsing.

The payoff is best felt through the long-context number. Serving a model at a 1-million-token context is brutally memory-hungry, because the model must hold a huge running cache of everything it has read. By shrinking the model and quantizing aggressively, Puzzle frees enough memory that a single H100 can juggle eight such long-context conversations at once instead of one. For anyone running an inference service, that is close to an eight-fold cost reduction on exactly the workloads -- long documents, big codebases, extended agent sessions -- that are otherwise the most expensive to serve.

Why it matters is that compression is where a lot of real-world AI value now gets unlocked. The models capturing headlines are the frontier ones; the models people can actually afford to deploy at scale are the compressed ones. A pipeline that reliably turns a 120B model into a 75B model at roughly double the throughput, with accuracy mostly intact, is the kind of unglamorous engineering that decides which capabilities reach production. The honest caveat is that the throughput and accuracy figures are Nvidia's own, measured under its chosen constraints, and 'retains strong downstream accuracy' is a summary, not a guarantee -- compression always costs something, and where exactly a 75B distillation is weaker than its 120B parent is the sort of thing that surfaces only once outside developers put it through their own workloads.

Primary source, verified: read the paper → (arXiv 2607.04371)

Ground Truth. (2026, July 11). Nvidia's 'Puzzle' pipeline shrinks a 120B model to 75B and roughly doubles serving throughput. Ground Truth. https://groundtruth.day/news/nemotron-puzzle-compresses-120b-model-to-75b.html

@misc{groundtruth:nemotron-puzzle-compresses-120b-model-to-75b,
  title  = {Nvidia's 'Puzzle' pipeline shrinks a 120B model to 75B and roughly doubles serving throughput},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/nemotron-puzzle-compresses-120b-model-to-75b.html}
}

Topics: model-compression · nvidia · efficiency · quantization · mamba