News · 2026-07-28

DeepSeek V4 Flash hits 32 tokens a second on one desktop

A published benchmark from Lucebox shows DeepSeek V4 Flash generating 32.0 tokens per second running entirely on one AMD Ryzen AI MAX+ 395 machine with 128 GB of unified memory, with no remote inference involved. The model is a 284-billion-parameter mixture-of-experts system with 13 billion active parameters per token. It fits and runs at that speed only because four separate systems techniques are stacked on top of each other.

Key facts

• 32.0 generated tokens per second, measured on a 2,048-token prompt, 510-token output, 8,192-token context, temperature 0, batch size 1.
• The model is 284B total / 13B active parameters, compressed to a 102.3 GB file at roughly 2.88 bits per parameter.
• Without speculative decoding the same setup runs at 25.31 tokens per second.
• Published by Lucebox; model card at Hugging Face.

The framing this result usually gets - "small models are catching up" - is wrong, and the correct framing is more interesting.

Nothing about this model is small. Its full weights are hundreds of gigabytes. What makes it locally practical is a stack of four techniques, none of which is new on its own, that compound when combined.

The first is mixture of experts. The model contains a large pool of specialist sub-networks, and each token is routed through only a few of them. So while the model holds 284 billion parameters, generating any given token involves about 13 billion. You pay for the whole library in memory, but only read a few books per query.

The second is quantization, pushed further than usual. Lucebox converted the model to a mixed ROCmFPX format in which the dominant routed-expert gate and up matrices use 2-bit blocks, expert down projections use 3-bit, and denser or more sensitive tensors keep 4-bit or higher precision. The result averages about 2.88 bits per parameter and lands at 102.3 GB - which is the entire reason it fits in 128 GB of unified memory at all. The point of the mixed scheme is that not all weights matter equally; the ones the model is most sensitive to keep their precision, and the bulk gets crushed.

The third is speculative decoding, here in the form of DSpark with a cap of four. A small fast draft model proposes up to three tokens ahead, and the big model verifies four positions in a single fused pass. When the draft guesses right, you get several tokens for the price of one forward pass. This is where a meaningful chunk of the headline comes from: the same target runs at 25.31 tokens per second in ordinary autoregressive mode.

The fourth is on the input side. Reading the prompt used indexed sparse prefill at roughly 245 tokens per second, with validation runs between 246.8 and 255.9. Lucebox is careful to note this is not exact prefill, and that exact prefill was much slower in its own checks - an approximation, honestly labeled.

One further change deserves the caveat treatment it gets in the source. Routing was cut from the model's default six experts per token to four. Lucebox states plainly that this trades quality margin for speed. So the 32 tokens per second figure describes a deliberately tuned configuration, not the model as its authors shipped it, and nobody has independently reproduced it.

Why this matters beyond one benchmark: it shows that the ceiling on local inference is now set by systems engineering rather than by model size. Each of these four techniques has been available for a while. Getting all four to cooperate - a 2-bit quantization scheme that a draft model can still speculate against, on a consumer APU with unified memory, using an approximate prefill path - is the actual work. See why LLM inference is memory bound for the underlying constraint all of this is fighting.

The result also arrived inside a visible same-day shift in the local-model community. Within hours on r/LocalLLaMA there was this benchmark, a post titled "a 5B-active model doesn't know much," an appreciation thread for Gemma 4 26B A4B (25.2 billion total parameters, 3.8 billion active), and a thread on Microsoft's new codec-native vision model. The common thread is a growing willingness to accept a model that knows fewer facts, on the theory that retrieval and tools can cover the gap.

That thesis is not settled, and the research is genuinely mixed. RGB evaluates how retrieval systems handle noise, irrelevant results and contradictory evidence. URAG shows accuracy gains collapsing under retrieval noise. HybridRAG-Bench exists specifically because separating genuine retrieval-and-reasoning from the model quietly recalling something it memorized is hard to measure. See retrieval-augmented generation.

The honest summary: the local-inference floor is falling because inference engineering compounds, and because active parameters, not total parameters, set the speed. Whether a model with thin parametric knowledge is genuinely fine when wired to good retrieval remains a community conviction rather than a demonstrated benchmark result.

See also: three ways the local inference floor fell and DeepSeek V4's million-token context by default.

Primary source, verified: read the paper → (arXiv 2606.19348)

Ground Truth. (2026, July 28). DeepSeek V4 Flash hits 32 tokens a second on one desktop. Ground Truth. https://groundtruth.day/news/deepseek-v4-flash-hits-32-tokens-a-second-on-one-desktop.html

@misc{groundtruth:deepseek-v4-flash-hits-32-tokens-a-second-on-one-desktop,
  title  = {DeepSeek V4 Flash hits 32 tokens a second on one desktop},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/deepseek-v4-flash-hits-32-tokens-a-second-on-one-desktop.html}
}

Topics: local-inference · quantization · mixture-of-experts · speculative-decoding · deepseek · open-weights