Learn · Intermediate

Queueing and tail latency: why a busy AI service suddenly feels slow

Queueing theory explains how jobs wait when they compete for limited service capacity. Tail latency describes the slow end of the resulting response-time distribution, such as the time by which 99% of requests finish. Together they explain why an AI service can generate tokens quickly in isolation yet feel slow and unpredictable under load.

Imagine a checkout that handles one customer per minute on average. At quiet times, a new customer is served immediately. Near capacity, one unusually large basket can create a line that takes a long time to disappear. The cashier has not become slower; the system has lost the spare capacity needed to absorb variation.

AI requests vary for similar reasons. Prompts have different lengths, answers contain different numbers of tokens, tools take uneven amounts of time and users arrive in bursts. Response time includes both the time a request waits and the time it receives service. Measuring only active computation leaves out the queue the user experiences.

A simple model shows the pressure

Consider an idealized single-server queue with independent random arrivals and exponentially distributed service times. If arrivals average eight jobs per second and the server averages ten jobs per second, utilization is 80%. In that specific model, average time in the system is one divided by the service rate minus the arrival rate: half a second.

Raise arrivals to nine jobs per second with the same server. Average time becomes one second. The server still averages a tenth of a second of work per job, but shrinking spare capacity doubles total response time. These are illustrative model calculations, not measurements of any AI provider. Real systems violate several of the model’s assumptions.

The useful point survives the simplification: high utilization leaves little room to recover from bursts and long jobs. Averages alone can hide that pressure. A server operating below capacity over a whole day can still saturate during the minutes that matter to customers.

Little’s law connects work and waiting

John D. C. Little’s 1961 paper establishes the relationship commonly written as L = λW. In a stable system with consistent boundaries, average work in the system equals the long-run arrival rate multiplied by average time spent there. Work includes both waiting and active jobs if time is measured from arrival through completion.

Suppose a stable service receives five requests per second and each spends two seconds inside it on average. It contains ten requests on average. This identity does not tell you the best scheduler or predict the entire latency distribution. It helps check whether throughput, concurrency and time measurements describe the same system.

Token streaming complicates the boundary. Time to the first token, time between tokens and time to the completed answer measure different experiences. A service can start quickly and finish slowly, or queue initially and then stream smoothly. Ground Truth’s prefill and decode lesson explains the distinct work behind those stages.

The tail matters when work fans out

Jeffrey Dean and Luiz André Barroso’s The Tail at Scale explains why occasional slow components become important in large services. An agent that depends on several searches or tools may wait for its slowest required result. Even fairly reliable components can produce a less predictable combined workflow.

As an illustration, if each of 100 independent parallel calls has a 99% chance of finishing before a threshold, the probability that all finish before it is about 37%. Independence is a strong assumption, and shared outages can make real behavior different. The calculation simply shows how many opportunities for delay accumulate when completion depends on every call.

Report percentiles with a defined workload and measurement window. The median describes the middle request; p99 describes a much slower boundary. Neither replaces the other. Averages remain useful for resource accounting, but a good average does not guarantee a usable interactive experience.

Scheduling changes the trade-off

Continuous batching keeps devices busy by mixing active requests. Waiting to form a larger batch can improve throughput while adding delay. Long prompts can obstruct short requests unless the scheduler separates or chunks their work. Admission control, additional capacity and workload-aware scheduling offer different ways to manage the queue.

DeepSeek’s deployment notes describe separate prompt-processing and generation pools with their own balancing requirements. Mistral’s voice latency documentation highlights another boundary: a transcription delay setting does not include every wait in a spoken agent task.

Measure realistic arrival bursts, request lengths and downstream dependencies before choosing an optimization. Track first response, completion time and slow-request percentiles alongside throughput. The goal is enough dependable service capacity to meet the intended experience, with queues visible in the measurements rather than hidden behind a fast isolated-model benchmark.

Ground Truth. (2026, October 21). Queueing and tail latency: why a busy AI service suddenly feels slow. Ground Truth. https://groundtruth.day/learn/queueing-and-tail-latency-in-ai-services.html

@misc{groundtruth:queueing-and-tail-latency-in-ai-services,
  title  = {Queueing and tail latency: why a busy AI service suddenly feels slow},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/queueing-and-tail-latency-in-ai-services.html}
}

Topics: fundamentals · queueing · latency · inference · systems