Learn · Intermediate

Continuous batching and paged attention: how one GPU serves a thousand people at once

Continuous batching and paged attention are the two scheduling techniques that let a single GPU serve hundreds of users simultaneously rather than one at a time. Neither changes what a model says. Both change what it costs to say it, and together they account for much of the order-of-magnitude fall in the price of running a large language model since 2022. If you have wondered how providers charge fractions of a cent for something that needs a data-centre GPU, this is most of the answer.

Both ideas come from systems research rather than machine learning: Orca, presented by Gyeong-In Yu and colleagues at OSDI in 2022, introduced continuous batching, and PagedAttention, from Woosuk Kwon and collaborators at Berkeley in 2023, introduced the memory scheme and the vLLM server built around it.

Why serving is hard

A GPU is enormously parallel. Running one request through it is like driving a bus with one passenger: the trip costs the same whether the bus is empty or full. So servers batch requests together, processing many at once.

The trouble is that text generation is not one operation. It is two, with different shapes — a distinction covered in prefill and decode. First the model reads your prompt, which it can do in parallel. Then it generates output one token at a time, each token depending on the last. Generation is inherently sequential, and each step uses very little of the GPU's arithmetic capacity. That is why inference is memory-bound: the machine spends its time moving weights around, not calculating.

Traditional static batching groups requests, runs them together, and returns the results when all are done. The flaw is obvious once stated. If one user asks for a haiku and another asks for an essay, the haiku finishes in twenty tokens and its slot in the batch sits empty for the next two thousand steps. With realistic traffic, most of the batch is idle most of the time.

Continuous batching: refill the seats every stop

Continuous batching — sometimes called in-flight batching — treats the batch as a rolling population rather than a fixed cohort. At every generation step, the scheduler retires any request that has finished and admits a waiting one in its place.

The bus becomes a city bus rather than a coach tour. Passengers get on and off at every stop, and the bus is full for the whole route. Nobody waits at the depot until a complete tour group assembles, and nobody rides to the end of the line because the slowest passenger has not arrived yet.

The gain is largest exactly where static batching hurts most: mixed workloads with wildly varying output lengths, which is what all real traffic looks like. Reported throughput improvements are often several-fold, and — this is the part that surprises people — latency usually improves too, because new requests start generating immediately instead of queueing for the next batch.

Paged attention: stop reserving the whole hotel

Continuous batching creates a second problem. Every active request needs its KV cache — the stored intermediate values for every token so far, which is what saves the model from recomputing the entire conversation at each step. That cache grows as the text grows, and with many requests in flight it dominates GPU memory.

The naive approach reserves one contiguous block per request, sized for the longest possible reply. This is catastrophic on two counts. If a request is allowed 4,000 tokens and uses 200, the other 3,800 slots are reserved and unusable. And because the blocks must be contiguous, memory fragments: you can have plenty free in total and still be unable to admit a request, like a restaurant with thirty empty seats but no table for six.

Paged attention borrows the fix that operating systems have used for decades — virtual memory. The cache is divided into small fixed-size blocks that can live anywhere in GPU memory. Each request keeps a table mapping its logical sequence of tokens to whatever physical blocks it has been given. Blocks are handed out only as tokens are actually generated.

The waste drops from most of the reserved memory to at most one partly filled block per request. The Berkeley team measured cache utilisation rising from roughly 20-40% to over 95%, and since the cache is the binding constraint on how many requests fit, near-tripling its efficiency near-triples the users a GPU can serve.

Paging brings a bonus that pure accounting would miss: blocks can be shared. If ten users hit the same long system prompt, that prefix lives in memory once and all ten point at it, with copy-on-write when their conversations diverge — the mechanism underneath prompt caching.

Why this is worth understanding

These techniques explain economics that otherwise look impossible. When a provider quotes a price per million tokens, that price assumes near-perfect batching. It is also why providers care so much about your traffic shape: predictable, cacheable, similar-length requests batch beautifully, while long and erratic ones do not, which is part of why token economics reward some usage patterns over others.

It also explains a live design pressure in new models. When DeepSeek compresses its attention cache to a few hundred bytes per token, or a lab adopts multi-head latent attention or sparse attention, the goal is not a better benchmark score. A smaller cache per request means more concurrent requests per GPU, which means a lower price per token. Architecture choices in 2026 are increasingly serving-economics choices wearing a research hat.

The honest limit: neither technique makes a single request faster. One user alone on a GPU sees no benefit at all. These are throughput optimisations — they improve what a machine achieves under load, which is the only condition production systems ever run in.

Ground Truth. (2026, September 10). Continuous batching and paged attention: how one GPU serves a thousand people at once. Ground Truth. https://groundtruth.day/learn/continuous-batching-and-paged-attention.html

@misc{groundtruth:continuous-batching-and-paged-attention,
  title  = {Continuous batching and paged attention: how one GPU serves a thousand people at once},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/continuous-batching-and-paged-attention.html}
}

Topics: inference · serving · kv-cache · efficiency · fundamentals · systems