How AI Models Answer Many People at Once
Follow an AI request from prompt to streamed answer, and learn how serving systems share GPU computation and memory through batching, KV caches, PagedAttention and phase separation. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio
Listen on Spotify · Watch on YouTube · Pocket Casts
How one model answers many people
Eris: <|prosody:expressive_high|>The PagedAttention researchers found that organizing conversation memory into reusable blocks let their AI serving system <|prosody:pause|>handle more simultaneous requests at similar latency to the systems they compared it with. Their tests used language models answering requests shaped like real conversations and instructions; the advantage depended on the workload and available memory.
Vestra: For each request, the system uses a trained language model. That model is a collection of learned numerical settings, called weights. Training adjusts those settings; inference is using the trained model to produce a result. A serving system receives a request from an application, prepares the input for the model, runs inference, and sends the answer back. It can send the answer in pieces as they become available.
Eris: Say I ask it to explain a lunar eclipse. My question is the prompt: the input the model uses to produce an answer. The server turns the text into tokens, small pieces of text represented as numbers. A token might be a whole short word, part of a longer word, or punctuation. The pieces are the units the model processes, not a fixed count of words. <|prosody:expressive_high|>But if my whole question is already there, why does the answer come out a piece at a time?
Vestra: <|prosody:expressive_high|>First the model processes the prompt. That stage is often called prefill. Because the prompt tokens are already available, much of their computation can happen in parallel, and the model can produce a first answer token. Then it repeatedly uses the preceding text, including what it has just generated, to predict a next token. That step-by-step process is called autoregressive generation, or decoding. It stops when the model produces a stop signal or reaches a limit.
Eris: So the next piece depends on the pieces before it. But the server may be doing this for a student asking about an eclipse, a programmer completing code, and someone summarizing a long document at the same time. <|emotion:surprise|>What strikes me is that their prompts arrive at different moments, and nobody knows in advance exactly when every answer will finish.
Vestra: The work commonly runs on a graphics processing unit, or GPU: a processor designed to perform many numerical operations at once. It has its own fast but limited memory. <|prosody:expressive_high|>The serving system's job is to decide which requests get that compute and memory at each moment, while returning the resulting text to the right application.
The memory behind each answer
Eris: <|prosody:expressive_high|>The model has to keep track of the conversation as it writes. Why doesn't it simply read the entire prompt and answer from scratch for every new token?
Vestra: It can recompute that information, but doing so repeatedly is expensive. In a common model design called a transformer, an operation called attention uses information from earlier tokens when processing the current one. The model computes numerical representations called keys and values for those earlier positions. They help it determine which earlier information to draw on and what information to combine. A key-value cache, usually shortened to KV cache, saves those intermediate representations so later steps can reuse them.
Eris: <|emotion:enthusiasm|>I like the reuse. But is that cache where the model keeps its learned facts?
Vestra: It is working memory for a particular sequence of text, <|prosody:pause|>not a database of facts the model has learned. The server still has to run the model and use the saved information at each step; caching avoids calculating those earlier keys and values all over again.
Eris: <|sfx:humming|>Hmm, so the memory bill grows while the conversation grows. Longer prompts start with more cached information, and generated tokens add more. The same GPU memory also holds the model's weights, which are shared across requests, plus temporary working data. Each active conversation needs its own state unless a portion can safely be shared.
Vestra: The PagedAttention paper examined systems that reserved a large continuous stretch of memory for a request, including room for an answer whose final length was unknown. <|prosody:expressive_high|>If the answer ended early, part of that reservation went unused. Different-sized reservations could also leave gaps that were awkward to reuse. With many active requests, wasted space meant fewer could fit together.
Eris: <|prosody:expressive_high|>That's frustrating: unused space, but fewer requests can fit. That makes memory management a capacity issue. Even if the GPU has arithmetic power left, a server cannot simply add another conversation if there is nowhere to keep its state. The PagedAttention researchers addressed that storage problem with a different way to allocate memory.
Sharing the GPU while reducing waits
Eris: <|prosody:expressive_high|>A serving system has two measurements to balance. Latency is how long an individual request waits: for example, from sending the prompt to seeing the first piece of the answer. Throughput is how much work the whole system completes in a given time, such as requests completed per second. A busy service can have high total throughput and <|prosody:pause|>still make one person wait too long. I'd care about that wait if I were the one asking.
Vestra: One way to improve throughput is batching: grouping work from several requests so the GPU processes it together. During generation, several conversations can each contribute their next step to a batch. The shared model weights can be used across that work, helping the GPU do more with a pass through its memory.
Eris: <|prosody:expressive_high|>But if the batch is fixed until everyone finishes, the short answers leave empty places. A new request might wait for the longest answer even though the others have already gone. Waiting to collect a bigger batch in the first place can also delay someone who arrived early. Does that group have to stay fixed?
Vestra: <|emotion:enthusiasm|>It can change as the model runs, with iteration-level scheduling, also called continuous batching. An iteration is one round of scheduled model work. Between rounds, the scheduler can remove finished requests and admit waiting ones, subject to compute and memory limits. A request needn't wait for every member of the original group to finish, though it can still queue when the system is full.
Eris: <|prosody:expressive_high|>But there is a second wait hiding inside a streamed answer: the gap between successive pieces. A long new prompt can interrupt answers already being generated, even though continuous batching is meant to keep the machine busy. How can a scheduler limit that interruption?
Vestra: The Sarathi-Serve researchers studied that problem. In the older serving systems they tested, admitting a long prompt could delay ongoing generation. <|emotion:enthusiasm|>Their scheduler breaks a long prompt into chunks and combines a chunk with steps from ongoing answers, within a limit on how much work it schedules at once. Existing answers keep advancing while the new prompt is processed over several rounds. Sarathi-Serve is a research prototype based on vLLM, not a feature-complete version of it.
Eris: In their tests with conversation-like requests and requests to summarize research papers, Sarathi-Serve sustained more incoming work than the comparison systems while meeting their chosen limits on long gaps between output tokens. <|emotion:relief|>That's the result I'd care about while watching text stream, not just while counting requests.
Vestra: <|prosody:expressive_high|>But I wouldn't call smaller chunks an automatic win. The size of the chunks matters. Smaller chunks can reduce interruptions but add overhead and may slow the new request's first answer token. The paper tested particular models, GPUs, request patterns and latency targets; it doesn't establish one best chunk size or a universal speedup. The current vLLM architecture documentation describes a token-budget scheduler that can also handle chunked prompt processing, so the precise behavior of any deployed version needs checking.
Making room with PagedAttention
Eris: <|prosody:expressive_high|>Continuous scheduling lets requests enter and leave, but their KV caches still need room as they grow. PagedAttention tackles that part of the problem by borrowing an idea from operating systems, the software that manages a computer's resources.
Vestra: An operating system can give a program the impression of continuous memory while placing fixed-size pages in different physical locations. PagedAttention applies a similar idea to the saved keys and values: it divides a request's cache into fixed-size blocks. A table records where the blocks belonging to that request actually live, and the attention calculation follows that table to find them.
Eris: So when those blocks fill up, does the cache need a bigger continuous stretch?
Vestra: <|emotion:enthusiasm|>Suppose a cache fills two blocks and then needs more space. The manager can allocate another available block wherever it fits, instead of reserving a huge unbroken stretch at the beginning. <|prosody:expressive_high|>The last block can still have unused space, and looking up scattered blocks adds work, but much less space needs to be held aside for an answer that may never be written.
Eris: <|emotion:enthusiasm|>And here's the part I like: the same mapping also makes sharing possible. If an application asks for two different answers to the same prompt, both answers can initially use the same saved prompt blocks. As the answers diverge, their new state is kept separately. Sharing depends on the context <|prosody:pause|>actually matching; a word appearing in two unrelated conversations is not enough.
Vestra: The researchers built the original vLLM serving system around this approach. In tests on models handling conversation and instruction workloads, it sustained more requests at similar measured latency than their comparison systems. One comparison was the authors' own implementation of an earlier system, because that system wasn't publicly available. <|prosody:expressive_high|>The gain was smaller in a tested case with short sequences and ample cache space, where computation became the limiting factor.
Eris: <|prosody:expressive_high|>That smaller gain is a useful check on my excitement. PagedAttention reorganizes the model's working memory. It doesn't teach the model a new fact or make it understand the question better. It can make room for more work, while the answer still comes from the same trained model.
Two phases, different hardware pressures
Eris: <|prosody:expressive_high|>Prompt processing and answer generation are connected, but they use the machine differently. That difference is why a serving system might schedule them carefully together, as Sarathi-Serve does, or put them on separate resources, as a system called DistServe proposes.
Vestra: During prefill, the prompt is already known, so the GPU can work on many prompt tokens in parallel. For a substantial prompt, the limiting factor is often how quickly the processor can do the arithmetic. During decoding, each request contributes only one new token at a time. Especially with a small batch, the limiting factor is often how fast the system can move the model's weights and cached state through memory to the processor.
Eris: <|prosody:expressive_high|>Those are tendencies, not labels stamped on every request. A large enough decoding batch can make arithmetic the limiting factor too, and the model, prompt length and hardware all matter. Still, the difference gives engineers two distinct jobs to schedule.
Vestra: <|emotion:enthusiasm|>DistServe assigns prefill and decoding to separate GPU resources. The prefill side processes the prompt and produces the first token; then it transfers the request's KV cache and first token to a decoding side that continues the answer. Each side has a copy of the model weights and can be provisioned and configured for its own job. Separating the work prevents a long prefill from directly occupying the GPU that is generating another answer. <|prosody:expressive_high|>That separation appeals to me. Does the handoff worry you?
Eris: <|sfx:humming|>Hmm, the handoff is the catch. The decoder needs the saved context, so the cache has to cross a connection between devices. <|prosody:expressive_high|>A longer prompt can mean more data to move. Separate copies of the weights and dedicated resources also consume memory and capacity, and either side can end up waiting if the other cannot keep up.
Vestra: The DistServe team compared its design with contemporary shared-resource serving systems on chatbot, code-completion and summarization workloads. They measured how many requests per GPU could meet their chosen targets for both the first token and subsequent generation. DistServe handled more under the paper's tested conditions. <|emotion:relief|>Its placement strategy kept corresponding stages close enough to use fast connections in the test cluster, making cache transfer a small part of measured latency there.
Eris: <|prosody:expressive_high|>That eases my concern about the handoff in their test cluster. It doesn't make separation automatically faster or cheaper. With slow connections, limited GPUs, different traffic, or a workload that can wait for complete answers, the transfer and extra model copies can outweigh the benefit. Sarathi-Serve's paper did not provide a direct quantitative comparison against disaggregated designs, so we can't rank the two from these papers alone.
From request to GPU workers
Eris: <|prosody:expressive_high|>vLLM is a real serving engine. Its documentation describes separate jobs for the part that talks to an application, the part that schedules work, and the processes that run the model.
Vestra: The application sends a request through an API, an agreed interface for programs to communicate. The API server receives it, prepares the input, including converting text to tokens, and sends it to the engine. The engine's scheduler decides which requests get work in each step. Its cache manager tracks the blocks needed for their saved attention information. GPU workers load and run the model, and results go back through the server to the application, potentially as a stream.
Eris: The scheduler and cache manager have to cooperate. If a running answer needs another block and memory is full, the system has to make a choice; changing the batch list alone can't create more memory. What happens to the answer that's already running?
Vestra: In the documented vLLM version, the scheduler can preempt a request: pause it, free its cache blocks and return it to the waiting queue. When admitted again, its progress can be recomputed, although cached prefixes or other mechanisms may restore some work. <|prosody:expressive_high|>That preserves the ability to continue, <|prosody:pause|>but recomputation uses time and compute. The original PagedAttention paper also explored moving state to ordinary system memory and bringing it back, which trades recomputation for transfer costs.
Eris: <|sfx:sigh|>Uh, having to redo that work is painful. And sometimes the weights themselves won't fit on a single GPU. The model can be split across several. With tensor parallelism, the devices share parts of the same layer's calculations. With pipeline parallelism, different devices hold successive groups of layers and pass intermediate results along. Both give the model more combined memory; both require communication. So what does that communication cost us?
Vestra: <|prosody:expressive_high|>Fast links matter especially when devices must exchange results repeatedly. Pipeline stages can also sit idle while waiting for work from an earlier stage. More GPUs are not a guarantee of a faster answer, and the best split depends on the hardware and workload. If one GPU has enough room, the vLLM scaling guide says distributing the model may be unnecessary.
Eris: <|emotion:determination|>That's why I'd want to see the waits, not just the total output. For a real service, check the time to the first token, the gaps between later tokens, total throughput, and how those change under the traffic and prompt lengths you actually expect. Also watch cache capacity and preemptions. An average can hide the people who wait much longer, and a larger batch can improve the system's total output while increasing an individual's delay.
Vestra: <|prosody:expressive_high|>Serving quality has two parts the user can feel: whether the answer is useful, and whether it arrives in time when other people are using the system too. Memory layout, scheduling, connections and available hardware determine how much of that model's capability the service can deliver at once.
Follow the research
Eris: <|prosody:expressive_high|>For today's AI headlines, listen to our separate news brief. You'll find the stories and sources at Ground Truth, groundtruth.day. <|emotion:enthusiasm|>Follow AI in Detail for more research explained.