Learn · Intermediate

Prefill and decode: why a model's first token and its next one are completely different problems

Running a language model is not one operation, it is two, and they stress a computer in opposite ways. Prefill reads your entire prompt in a single parallel pass and is limited by how fast the chip can do arithmetic. Decode generates the reply one token at a time and is limited almost entirely by how fast the chip can read memory. Nearly every practical fact about AI serving costs, latency, and hardware choice follows from that split.

The two phases

When you send a prompt, the model first has to read it. Every token in the prompt gets processed through every layer, and because all of those tokens are already known, the whole thing can happen at once as large matrix multiplications. This is prefill. It builds the model's internal working state for your prompt - the KV cache, which stores what each layer computed for each position so the model never has to recompute it.

Prefill ends the moment the model emits its first token. Everything after that is decode.

Decode is a different animal. Token 51 depends on token 50, which depends on token 49. There is no way to compute them in parallel, because each one has to exist before the next can be predicted. So the model runs a full forward pass through every layer to produce exactly one token, appends it to the KV cache, and does it again.

Why the second phase is so much worse

Here is the part that surprises people: decode does almost no useful math.

To generate one token, the model must move its active weights from memory into the compute units. On a large model that is tens of gigabytes of traffic. Then it multiplies those weights against a single token's worth of data - a vector, not a matrix. The chip's arithmetic units, capable of trillions of operations per second, spend nearly all their time idle, waiting for bytes to arrive.

Think of a chef with a huge kitchen, cooking one grain of rice at a time. Each grain requires fetching every ingredient from the pantry. The chef's knife skills are irrelevant; the walk to the pantry is the whole job. That walk is memory bandwidth, and it is why LLM inference is memory-bound.

The rule of thumb that falls out: peak decode speed is roughly memory bandwidth divided by bytes touched per token. On a machine with 1.2 terabytes per second of bandwidth, a model whose active weights are 40 GB cannot exceed about thirty tokens per second no matter how much compute is attached. This is why Apple's Mac Studio holding 512GB of memory does not make big models fast, and why quantization - shrinking each weight to fewer bytes - speeds up generation even though it does not reduce the amount of arithmetic.

Prefill has the opposite profile. A thousand-token prompt means every weight fetched from memory gets multiplied against a thousand tokens instead of one, so the same memory traffic buys a thousand times more work. Prefill saturates the math units. It is the only phase where a chip's headline compute number means much.

What this explains

Two different latency numbers. Time to first token is a prefill measurement, and it scales with prompt length. Tokens per second afterwards is a decode measurement, and it barely depends on prompt length at all. A system can be excellent at one and terrible at the other, which is why serving benchmarks that report a single "speed" figure are close to meaningless.

Why input tokens are cheaper than output tokens. Look at any provider's pricing and output costs several times more than input. That is not margin strategy, it is the physics above: an input token gets processed in a batch of thousands, an output token gets a whole forward pass to itself. See inference cost and token economics.

Why prompt caching works so well. If prefill is expensive and its only product is the KV cache, then storing that cache and reusing it for a repeated prefix skips the expensive phase entirely. That is exactly what prompt caching does.

Why batching helps decode enormously. If you generate for fifty users simultaneously, you fetch the weights once and use them fifty times. Decode throughput scales almost linearly with batch size until memory runs out, which is why serving many users is dramatically cheaper per token than serving one.

What people build because of it

Once you see the two phases as different workloads, obvious engineering follows.

Chunked prefill, introduced in SARATHI by Amey Agrawal and colleagues, splits a long prompt into pieces and interleaves them with other users' decode steps. Otherwise one person pasting a long document freezes everyone else's generation - a problem large enough to have its own name, head-of-line blocking.

Disaggregation goes further and puts the phases on separate machines entirely. Splitwise from Microsoft Research and DistServe from Yinmin Zhong and collaborators both showed that dedicating one hardware pool to prefill and another to decode raises useful throughput while meeting latency targets, because you can then buy compute-heavy machines for one and bandwidth-heavy machines for the other.

Memory management for the cache itself. The KV cache grows with every token and every concurrent user, and naive allocation wastes most of it to fragmentation. PagedAttention by Woosuk Kwon and colleagues, the technique behind vLLM, borrows virtual memory paging from operating systems to fix this - one of the largest practical serving wins of the last few years.

Speculative decoding attacks decode's sequential nature directly: let a small fast model guess several tokens ahead, then have the big model verify all the guesses in one pass. Verification is prefill-shaped, which is the phase hardware is good at. When the guesses are right, you get several tokens for roughly the price of one.

The one-line version

Prefill is a compute problem you solve with better math throughput. Decode is a plumbing problem you solve with more bandwidth, smaller weights, and bigger batches. When someone quotes you a speed, a price, or a GPU recommendation, the first question is always: which phase are we talking about?

Ground Truth. (2026, August 25). Prefill and decode: why a model's first token and its next one are completely different problems. Ground Truth. https://groundtruth.day/learn/prefill-and-decode.html

@misc{groundtruth:prefill-and-decode,
  title  = {Prefill and decode: why a model's first token and its next one are completely different problems},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/prefill-and-decode.html}
}

Topics: inference · serving · memory-bandwidth · kv-cache · latency · throughput