Learn · Intermediate

Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math

Generating text with a language model is a memory problem, not a math problem. To produce a single word, the processor has to read every weight it needs out of memory, use each one for exactly one multiplication, and throw it away - so the chip spends most of its time waiting for data to arrive rather than computing. This single fact explains most of what looks strange about AI hardware: why a $30,000 accelerator runs at a small fraction of its advertised speed, why serving many users at once is nearly free compared with serving one, and why quantization delivers speedups far larger than the arithmetic savings would suggest.

The ratio that decides everything

Every computation has a ratio worth measuring: how many arithmetic operations you perform for each byte you move from memory. Researchers call this arithmetic intensity. A modern accelerator can do on the order of a hundred to a few hundred floating-point operations in the time it takes to fetch a single byte from its high-bandwidth memory. So if your computation does fewer operations per byte than that ratio, you are memory-bound and the arithmetic units idle. If it does more, you are compute-bound and the memory system keeps up.

This framing comes from the roofline model, and it is the most useful mental tool in performance engineering: plot arithmetic intensity on one axis and achievable speed on the other, and you get a line that rises with intensity and then flattens into a ceiling set by the chip's raw math throughput. Almost everything about transformer inference sits on the rising part of that line, nowhere near the ceiling.

Now do the arithmetic for text generation. When a model generates one token, it multiplies a single vector by each weight matrix in turn. A matrix-vector multiply uses each weight exactly once. One weight loaded, two operations performed (a multiply and an add). An arithmetic intensity of about two, against a machine that needs a hundred or more to stay busy. You are running at roughly two percent of the hardware's arithmetic capability, and no amount of code optimisation fixes it, because the bottleneck is physics: the weights have to physically travel from memory to the processor, and the wire is the slow part.

Why training looks completely different

Training the same model on the same hardware routinely hits a large fraction of peak arithmetic throughput. The difference is batching. During training you push thousands of token positions through the model simultaneously, so when you load a weight matrix you multiply it against a thousand vectors instead of one. Each loaded byte now does a thousand times more work, and arithmetic intensity climbs from two into the hundreds. The matrix-vector multiply becomes a matrix-matrix multiply, which is the operation these chips were designed for.

An analogy: imagine a warehouse worker fetching parts from a distant shelf. If the assembly line needs one bolt at a time, the worker spends the whole shift walking. If the line batches a thousand bolts per trip, the walk amortises to nothing and the worker is limited only by how fast they can hand bolts over. Training is the thousand-bolt trip. Generating a token for one user is the single-bolt trip.

This is why the economics of serving models are what they are. A provider batching hundreds of concurrent requests through the same weight loads gets close to compute-bound and can charge cents per million tokens. Someone running the same model locally for themselves is paying full price for every trip to the shelf. It is also why training and inference have such different hardware profiles despite running the same network.

What follows from it

Once you see the bottleneck, a long list of otherwise-unrelated techniques snaps into focus as the same idea.

Quantization is the most direct. Halving the precision of the weights halves the bytes you must move, and in a memory-bound regime that roughly doubles your speed - a bigger win than the arithmetic saving alone would predict. This is why 4-bit models feel disproportionately fast.

The KV cache exists to avoid recomputation, but it introduces its own bandwidth problem, because it must be read in full for every generated token and it grows with conversation length. That is precisely why so much recent architecture work - linear attention, sparse attention, bounded-state designs - aims at holding that cache to a fixed size instead of letting it expand.

FlashAttention is a pure data-movement optimisation: it computes exactly the same attention result, but reorganises the work so intermediate values stay in fast on-chip memory rather than round-tripping to slower memory. Nothing about the math changed; only the movement did.

Speculative decoding is the cleverest exploitation of the imbalance. A small model drafts several tokens ahead, then the large model verifies all of them in a single pass. Verifying five tokens costs almost exactly as much as generating one, because you were memory-bound anyway and the arithmetic units had capacity to spare. You get free work out of the slack.

Mixture-of-experts models are the same insight applied to model design: hold enormous total capacity in memory but touch only a fraction of it per token.

Seeing it in the wild

This year produced an unusually clear demonstration on unusually small hardware. A developer fit a 28.9-million-parameter model onto an $8 microcontroller with 512KB of fast memory by putting 25 million of those parameters in flash storage and reading only about 450 bytes of them per generated word. The measured cost of those flash reads was around 0.12 milliseconds per token. The expensive step, at roughly 17 milliseconds, was scanning the model's output layer out of slower external memory - a step involving far fewer parameters. Capacity was nearly free; movement was the bill. Precisely the same trade governs a rack of accelerators costing a hundred thousand times more.

The practical takeaway for anyone choosing hardware or optimising a deployment: look at memory bandwidth first, model size second, and arithmetic throughput last. And if your inference feels slow, ask how many bytes each generated token requires before you ask how many operations it requires. The answer is usually the first number.

Ground Truth. (2026, July 26). Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math. Ground Truth. https://groundtruth.day/learn/why-llm-inference-is-memory-bound.html

@misc{groundtruth:why-llm-inference-is-memory-bound,
  title  = {Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/why-llm-inference-is-memory-bound.html}
}

Topics: inference · hardware · efficiency · memory-bandwidth · fundamentals