Learn · Intermediate

Multi-head latent attention: compressing the memory that inference actually runs out of

Multi-head latent attention is an architecture change that shrinks the memory a language model needs while generating text, by compressing the keys and values it must remember into a single small shared vector rather than storing a separate pair for every attention head. DeepSeek introduced it in DeepSeek-V2 and reported cutting the key-value cache by more than 90% while matching the quality of standard multi-head attention. It targets the single largest memory cost in long-context inference.

The problem starts with how generation works. When a model writes text, it produces one token at a time, and each new token attends to every token before it. Recomputing the attention keys and values for the whole history at every step would be absurdly wasteful, so models store them -- the KV cache. The cache holds one key vector and one value vector per token, per attention head, per layer.

Multiply that out and the number gets alarming fast. A large model with dozens of layers and dozens of heads per layer, generating over a long context, can end up with a cache larger than the model's own weights. And unlike weights, which are loaded once and shared across every request in a batch, the cache is per-conversation: serve a hundred users at once and you need a hundred caches. This is why serving costs scale with context length in a way that surprises people, and why inference is memory-bound rather than compute-bound.

The field's first answers reduced the cache by sharing. Noam Shazeer's multi-query attention went to the extreme: keep all the query heads, but have them all share a single key-value head. The cache shrinks by a factor equal to the head count, and quality suffers, because each head was previously free to attend to a different aspect of the context and now they all read the same summary. Ainslie and colleagues' grouped-query attention split the difference -- groups of query heads share a key-value head -- and became the default in most modern open models precisely because it is a reasonable compromise.

Both are lossy in the same direction: they solve a storage problem by deleting distinctions.

Multi-head latent attention takes a different route. Instead of making heads share, it compresses. Keys and values are projected down into a single small latent vector per token, and that is the only thing stored. When attention is computed, each head reconstructs the key and value detail it needs by projecting back up from the shared latent representation with its own learned matrices.

The analogy: grouped-query attention is a team of specialists forced to work from one shared summary document. Multi-head latent attention gives them a compact shorthand encoding of the original material, from which each specialist expands the parts relevant to their own concern. The stored artifact is small either way. The second one preserves much more of what each specialist needed.

The critical detail is that the compression is learned during training, not bolted on afterwards. The model discovers, over the course of training, what information the down-projection has to preserve so the up-projections can do their jobs. That is why the quality loss is so much smaller than the compression ratio would suggest -- it is a trained bottleneck, not a truncation, the same principle that makes autoencoders work.

There is a practical subtlety that took some engineering to resolve. Rotary positional encoding, which most modern models use to tell tokens where they sit in a sequence, rotates the keys in a position-dependent way -- and that interacts badly with a compressed representation, because the rotation has to happen in the space where the keys actually live. DeepSeek's solution splits each head's dimensions into a compressed portion and a small uncompressed portion that carries the positional rotation. It is inelegant, and it works, and the DeepSeek-V3 technical report documents the approach at scale in a model combining this attention with a large mixture-of-experts design.

Why it matters is straightforward economics. Cache size determines how many conversations a GPU can hold at once, which determines cost per user. A ten-fold cache reduction means roughly ten times the concurrent sessions on the same hardware, or ten times the context length for the same memory. That is not an incremental win; it changes what is affordable to deploy.

It also fits a pattern worth noticing. The efficiency advances that actually stick -- FlashAttention, grouped-query attention, speculative decoding, this -- are almost never about doing less thinking. They are about moving less data. And the pressure keeps producing new variants: IFM's recently released K2 Horizon family introduced Mixture-of-Value Attention, which pushes sparsity into the attention layers themselves rather than compressing what they store, a different attack on the same wall.

The honest caveat: multi-head latent attention is an architectural decision, which means it must be trained in. You cannot convert an existing model to it the way you can quantize one after the fact -- grouped-query attention can at least be distilled from a multi-head checkpoint, which is part of why it spread faster. And the reported quality parity comes from the lab that designed it, on its own models. The technique is well regarded and increasingly copied, but independent head-to-head comparisons at matched scale remain thinner than the enthusiasm around it.

Ground Truth. (2026, September 3). Multi-head latent attention: compressing the memory that inference actually runs out of. Ground Truth. https://groundtruth.day/learn/multi-head-latent-attention.html

@misc{groundtruth:multi-head-latent-attention,
  title  = {Multi-head latent attention: compressing the memory that inference actually runs out of},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/multi-head-latent-attention.html}
}

Topics: attention · kv-cache · inference · architecture · deepseek · efficiency