Learn · Intermediate

FlashAttention: making attention fast by respecting the memory hierarchy

FlashAttention is an exact algorithm for computing attention - the core operation inside every transformer - that runs several times faster and uses dramatically less memory than the standard approach. It achieves this not by changing the math but by reorganizing it to respect how GPUs actually work, keeping the computation inside the chip's small, fast memory and never writing the enormous attention matrix out to slow memory. Introduced by Tri Dao and colleagues in 2022, it became the default attention implementation in modern models almost overnight.

To see why it matters, you need to know what attention costs. In a transformer, attention lets every token look at every other token to decide what is relevant (our lesson on transformers covers the mechanism). For a sequence of length N, that means computing an N-by-N grid of scores - how much each token attends to each other token. If your sequence is 8,000 tokens long, that grid has 64 million entries. The standard implementation computes this whole grid, writes it to the GPU's main memory, reads it back to apply a softmax, writes it again, reads it again to multiply by the values, and so on. The arithmetic is not the problem. The problem is all that reading and writing to slow memory.

This is the key insight, and it is a systems insight, not a math one. A modern GPU has two kinds of memory: a large pool of relatively slow 'high-bandwidth memory' (HBM), and a tiny pool of extremely fast on-chip memory (SRAM) - think of HBM as a big warehouse across the parking lot and SRAM as the small workbench right in front of you. Moving data between the warehouse and the workbench is the expensive part. Standard attention keeps hauling the giant attention matrix back and forth to the warehouse. FlashAttention's authors called their approach 'IO-aware' because it is designed around minimizing exactly this data movement.

The trick is tiling. Instead of computing the entire N-by-N matrix at once, FlashAttention breaks the problem into small blocks that fit on the workbench (SRAM). It loads a block of queries, a block of keys and values, computes the partial attention for just that block, and accumulates the running result - all without ever materializing the full matrix in slow memory. The clever part is the softmax: normally softmax needs to see a whole row of scores at once to normalize it, which seems to require the full matrix. FlashAttention uses an 'online softmax' technique that updates the normalization incrementally as it processes each block, mathematically reconstructing the exact same answer the all-at-once version would have produced. That is why it is exact: the output is bit-for-bit the attention you wanted, just computed in a memory-friendly order.

The analogy: imagine summing a million numbers written across a thousand pages in the warehouse. The naive way is to carry all thousand pages to your desk at once - except your desk only holds ten pages, so you can't. The FlashAttention way is to fetch ten pages, add them to a running total on a sticky note, return them, fetch the next ten, and keep updating the total. You never need all thousand pages on your desk, and you get the exact same sum. The running total is the online softmax; the ten-page desk is SRAM.

The payoff is large and practical. FlashAttention made attention roughly two-to-four times faster in practice and, crucially, cut its memory use from growing with the square of the sequence length to growing only linearly. That linear memory scaling is what made long context windows feasible - training and running models on far longer sequences without the attention matrix blowing up your GPU. It pairs naturally with the KV cache, another memory-management trick that makes generation efficient, and it sits alongside sparse attention, which takes a different route - skipping some token pairs entirely rather than computing all of them more efficiently.

FlashAttention is also why kernel-level optimization has become a frontier of its own. The lineage continued with FlashAttention-2, which improved how the work is split across the GPU's parallel units, and the pattern keeps advancing as hardware changes - the recent community buzz around FP4 attention kernels claiming speedups over FlashAttention-4 on next-generation chips is the same story continuing: as precision drops to 4-bit formats, the attention kernel gets rewritten again to exploit the new hardware. The enduring lesson of FlashAttention is that on modern accelerators, how you move data through the memory hierarchy often matters more than how many operations you do - and a smarter memory schedule, with identical math, can be the difference between a model that fits and one that doesn't.

Ground Truth. (2026, July 13). FlashAttention: making attention fast by respecting the memory hierarchy. Ground Truth. https://groundtruth.day/learn/flashattention.html

@misc{groundtruth:flashattention,
  title  = {FlashAttention: making attention fast by respecting the memory hierarchy},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/flashattention.html}
}

Topics: attention · transformers · gpu · efficiency · systems