Learn · Intermediate

Linear Attention

Linear attention is a family of techniques that rewrite the transformer's attention operation so its cost grows linearly with the length of the input rather than quadratically. In plain terms, standard attention gets dramatically more expensive as text gets longer, and linear attention makes that cost grow in a gentle straight line instead, which is what lets a model read a million-token document without melting. The trade is that it replaces perfect recall of every past token with a compact, running summary, so it is faster and lighter but remembers the past less precisely. This is the idea behind the Kimi Delta Attention layers in Moonshot's new Kimi K3, and behind a decade of work on making long-context models practical.

To see why it matters, you have to understand the problem with ordinary attention, the mechanism at the heart of the transformer. In standard attention, every token looks at every other token to decide what to pay attention to. If you have a hundred tokens, that is ten thousand comparisons; if you have a thousand tokens, that is a million. The cost scales with the square of the sequence length, so doubling the input roughly quadruples the work and the memory. That quadratic wall is the single biggest reason long context is expensive, and it is why techniques like the KV cache, FlashAttention, and sparse attention exist to soften it.

Linear attention attacks the problem at the root by changing the math. The key insight, introduced in the 2020 paper by Angelos Katharopoulos and colleagues aptly titled Transformers are RNNs, is that if you replace the softmax in attention with a simpler similarity function, you can reorder the computation. Instead of comparing every token to every other token, the model maintains a single running state, a fixed-size matrix that summarizes everything it has read so far, and updates that state one token at a time. Each new token folds its information into the summary and reads from it, so the work per token is constant and the total work grows linearly with length.

An analogy helps. Standard attention is like a researcher who, for every new sentence, re-reads the entire document from the beginning to decide what is relevant. Linear attention is like a researcher who keeps a running set of notes: each new sentence updates the notes, and they consult only the notes, never the full document again. The note-taker is enormously faster on long documents, but the notes are a lossy compression, so if you ask about one exact word buried on page 300, the full-document reader can find it and the note-taker may only have a fuzzy summary. That lossy-memory trade is the central limitation of linear attention, and it is why these methods historically lagged on tasks that demand exact recall.

The Katharopoulos formulation also revealed a deep connection: a linear-attention transformer processed one token at a time is mathematically a recurrent network, which is why the running-state view works and why linear attention sits so close to modern state space models like Mamba. A parallel line of work, the Performers paper by Krzysztof Choromanski and colleagues, reached linear cost a different way, by approximating the softmax with random features. More recent work, such as the DeltaNet line by Songlin Yang and colleagues, sharpens the update rule so the running state edits its memory more intelligently, using what is called the delta rule to overwrite stale information rather than just accumulating it, which is the lineage behind the Delta Attention now appearing in production models like Kimi K3.

Why it matters today: as models push toward million-token context windows, the quadratic cost of standard attention becomes the dominant expense, and pure linear attention loses too much precision to be used alone. The practical answer, visible across current frontier models, is hybrid architectures that interleave cheap linear-attention layers with a smaller number of full-attention layers, capturing most of the speed and memory savings while keeping enough exact recall to stay accurate. Linear attention is thus not a wholesale replacement for attention but a crucial tool in the efficiency toolkit that makes long context economically feasible.

The honest caveat: the field has repeatedly produced linear-attention variants that look great on speed benchmarks but underperform standard attention on tasks requiring precise long-range retrieval, and the gap has narrowed rather than closed. When a model vendor reports a new linear-attention layer, the question to ask is not just how fast it is, but how much recall it gives up, and whether it is being used pure or in a hybrid stack. Speed is the easy part; keeping the memory faithful is the hard part.

Ground Truth. (2026, July 22). Linear Attention. Ground Truth. https://groundtruth.day/learn/linear-attention.html

@misc{groundtruth:linear-attention,
  title  = {Linear Attention},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/linear-attention.html}
}

Topics: attention · efficiency · long-context · architectures