kv-cache
Continuous batching and paged attention: how one GPU serves a thousand people at once Lesson
Continuous batching and paged attention are the two scheduling tricks that let a single GPU serve many users at once instead of one at a time, and together they are responsible for most of the collapse in the price of running a language model.
Multi-head latent attention: compressing the memory that inference actually runs out of Lesson
Multi-head latent attention compresses the key-value cache that language models must hold in GPU memory during generation, projecting keys and values into a small shared latent vector instead of storing them per head. DeepSeek introduced it in DeepSeek-V2, reporting a cache reduction of more than 90% with quality matching full attention.
Prefill and decode: why a model's first token and its next one are completely different problems Lesson
Running a language model has two phases with opposite hardware profiles: prefill reads your whole prompt at once and saturates the chip's math units, while decode produces one token at a time and is limited almost entirely by memory bandwidth.
Inference cost and token economics: why output tokens cost more than input Lesson
Model providers charge separately for the tokens you send and the tokens the model writes, and output is typically three to five times more expensive. The reason is architectural: input is processed in one parallel pass, while every output token requires its own full pass through the model.
Attention sinks: the token that soaks up attention and means nothing Lesson
An attention sink is a token, almost always at the very start of a sequence, that absorbs a large share of a model's attention regardless of what it says, and it exists because softmax forces attention to add up to one whether or not anything deserves it.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
Three separate tricks dropped the hardware floor for local AI in one day News
A 26-billion-parameter Gemma model ran on an iPhone by streaming expert weights from storage, AMD shipped a sparse mixture-of-experts model activating 2.8 billion parameters per token, and a llama.cpp fork began saving conversation caches to disk - three unrelated attacks on three different bottlenecks.
Grafting a Verified Solution Cache Lets a Frozen Model Skip Fine-Tuning Entirely News
A new paper shows a frozen small language model's accuracy on a hard math test can jump from 80.0% to 93.3% simply by grafting a byte-exact cache of verified solutions into it, with no weight changes at all.
Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies Lesson
Prompt caching lets an AI provider store the processed form of a repeated chunk of text -- like a long system prompt -- so it can be reused across requests at a fraction of the cost, instead of being re-processed every time.
New methods page an AI's memory in and out like an operating system News
Two new techniques treat a language model's long-context memory like an operating system's memory hierarchy - keeping coarse summaries on the GPU and paging compressed detail out to the CPU - with one, SeKV, cutting GPU memory use by 53% at 128,000 tokens while avoiding the errors that plague simpler methods.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
CachyLLama Tool
MIT-licensed llama.cpp fork that saves conversation and system-prompt caches to SSD and restores them after a restart, so local agents stop reprocessing the same prompt prefix every turn. Its own benchmark reports long repeated agent prefixes going from minutes cold to about a second warm.