News · 2026-08-02

Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks

A community experiment on DeepSeek V4 Flash found that switching llama.cpp's key-value cache from full precision to its generic 8-bit format left average perplexity almost unchanged while materially changing the model's actual token choices, with the shortlist of plausible next tokens matching only about seven-eighths of the time. A comparison model showed dramatically less sensitivity to the same change. The finding is real and specific, and the headline it travelled under is wrong: DeepSeek's own serving recipe for this model uses a low-precision cache.

Key facts

• The test: full-precision versus llama.cpp Q8_0 key-value cache, on a heavily compressed V4 Flash quantisation at a short 512-token context.
• The anchor number: the set of tokens surviving the model's probability cutoff matched between the two runs only about seven times in eight, while average perplexity barely moved.
• When: posted 2 August 2026 to the LocalLLaMA community, with the method and raw output published.
• Primary source: the original experiment thread, with the counter-argument in DeepSeek's official 0731 model card.

To follow this you need one idea: the KV cache. As a language model reads and writes text, it stores a compressed record of everything it has already seen so it does not have to re-read the whole conversation for every new word. That store grows with the conversation and is often the largest thing in memory, so shrinking it by storing the numbers at lower precision is one of the standard levers local users pull. It is the memory equivalent of taking notes in shorthand.

The experimenter did something more careful than the usual perplexity check. Perplexity is an average across the entire vocabulary, and it is dominated by the enormous number of tokens the model was never going to pick anyway. It is a blunt instrument. Instead, the test compared the rankings and probabilities of the candidates that actually matter at each step. Average perplexity barely moved. The distribution of choices developed a fat tail: many rankings shifted, and the shortlist of tokens that survive the model's probability cutoff agreed only about seven-eighths of the time. In other words, roughly one step in eight, the shorthand notes changed which words the model considered saying at all.

Why would this model be unusually sensitive? Because its cache is not a plain uniform ledger. According to DeepSeek's architecture report, V4 Flash interleaves several mechanisms: overlapping compressed blocks of cached context, with a component DeepSeek calls a Lightning Indexer choosing which of those blocks to attend to; a second, far more aggressive path compressing at roughly 128 to 1; and a separate sliding-window tail state with its own update rules. The crucial word is choosing. In conventional attention, a small numerical error slightly blurs how much weight a memory gets. Here, a small error can change which block the model retrieves at all — a discrete decision, not a continuous one. It is the difference between misreading a word in your notes and turning to the wrong page. Once you are on the wrong page, everything after it compounds. This is a distinctive property of the sparse attention family V4 Flash belongs to.

That mechanism is persuasive. The general conclusion drawn from it is not. DeepSeek's official 0731 serving recipe explicitly uses an FP8 key-value cache together with an FP4 indexer cache, and the model was trained with quantization-aware treatment of the indexer's query-key path. The model was built to run a low-precision cache. What it was not built for is llama.cpp's generic Q8_0, a general-purpose format that applies its own block-wise scaling on top of representations that have already been compressed and scaled once. A commenter in the same thread makes exactly this point: these are not interchangeable formats. It is the same lesson that shows up everywhere in quantization — a model trained to tolerate a specific low-precision format behaves nothing like the same model with a different format bolted on afterwards.

There is an irony in the whole exercise. V4 Flash's compressed attention has already made cache memory unusually cheap. vLLM estimates its full-precision million-token cache at under 10 GiB, roughly halving under the intended FP8 and FP4 layout. So cache quantization is a poor first lever on this particular model. You are accepting a real risk of changed behaviour to reclaim memory the architecture already gave you.

The safe operating advice: avoid llama.cpp's generic 8-bit KV cache for V4 Flash until the runtime supports the model's own FP8 and indexer handling, and until someone has evaluated it on long agent tasks rather than a 512-token snippet. Do not repeat the claim that the model cannot use a low-precision cache.

The honest caveat cuts both ways. This is one experimenter's test, unreplicated, at a short context, on a heavily compressed quantisation of the model — exactly the configuration where numerical fragility is most likely to show. The shortlist-disagreement figure is a measurement of difference, not of damage: nobody has yet shown that the changed tokens produce worse answers on real work. That test is the obvious next thing someone should run.

Primary source, verified: read the paper →

Ground Truth. (2026, August 2). Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks. Ground Truth. https://groundtruth.day/news/quantizing-v4-flashs-kv-cache-in-llama-cpp-changes-which-tokens-it-picks.html

@misc{groundtruth:quantizing-v4-flashs-kv-cache-in-llama-cpp-changes-which-tokens-it-picks,
  title  = {Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/quantizing-v4-flashs-kv-cache-in-llama-cpp-changes-which-tokens-it-picks.html}
}

Topics: deepseek · kv-cache · quantization · local-ai · llama-cpp · inference