News · 2026-08-02

A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill

An open-source inference engine called Mference runs DeepSeek V4 Flash, a 284-billion-parameter model, on a 24-gigabyte Mac with an effective memory footprint of about three gigabytes. It does this by keeping only the parts of the model every token needs in memory and reading the rest off the SSD as each token requires it. The headline that spread — "284B in 5.3 GB" — describes the working set, not the download: the checkpoint still occupies 90 to 98 gigabytes of disk.

Key facts

• The measurement: about 3 GB effective footprint at the smallest expert-cache setting, or 5.9 GB at the setting that reaches the project's best short-run decode speed.
• The disk bill: roughly 90 to 98 GB, using an MLX checkpoint with routed experts at two bits and most core tensors at four.
• The test conditions: a 4,000-token context on a 24 GB M5 Mac, with roughly 2 GB of expert reads required per generated token when nothing is cached.
• Primary source: the Mference repository and its memory-budget document, which publishes all of the above.

The trick rests on how modern large models are actually built. DeepSeek V4 Flash is a mixture of experts: rather than one enormous network where every parameter fires for every word, it contains a large collection of specialist sub-networks, and a routing component picks a small handful of them for each token. The full model is vast. The part that runs at any one instant is not.

Mference exploits that gap directly. It keeps the shared core, the attention machinery and the growing conversation cache resident in memory — those are needed for every token, so they have to be there. Everything else, the routed experts, stays on the SSD. When a token arrives and the router picks its specialists, the engine fetches exactly those from disk and runs them. The library analogy is close to exact: you keep the reference desk and the card catalogue on your table, and walk to the stacks for each individual book. The desk is small. The library is not.

That is why the two numbers are both true and describe different things. The working set is genuinely about three gigabytes at the tightest setting. The library is still 90 to 98 gigabytes of shelving you must own. And the walking has a cost the headline omits: at zero cache hits, the design needs roughly two gigabytes of expert reads per generated token. Sustained generation therefore falls below the headline peak, because the SSD, not the processor, sets the pace. The project's documentation says this plainly.

Two further limits matter before anyone treats this as a deployment. The published test ran at a 4,000-token context, which is short — and long context is precisely the workload where the cache grows and the memory budget stops being comfortable. And the project states it has no token-level divergence or logit-parity test against the reference implementation. That is an unusually honest disclosure, and it draws the right line: the engine demonstrably produces output, and nobody has yet demonstrated that the output matches what the real model would say.

The result belongs to a growing genre. Last week brought a 26-billion-parameter model running in 2 GB by streaming experts off the SSD, and the same week produced a C99 engine running Kimi K3 inside 8 gigabytes of RAM at 33 seconds per token. Read together, these projects are making one argument from three directions: the binding constraint on local AI has moved. It used to be whether the weights fit in memory. For sparse models it is now whether your storage can feed the experts fast enough, and whether your runtime honours the model's particular structure — the same lesson visible in the week's other V4 Flash finding, that quantizing its cache in llama.cpp changes which tokens it picks.

The caveat is the one the project itself raises. This is a self-reported measurement from a single developer, on one machine, at one context length, without a quality parity check. It is an excellent systems demonstration and a genuinely interesting piece of engineering. It is not evidence that a five-gigabyte Mac configuration preserves the model's ability to do the long, multi-step agent work the model was actually built for.

Primary source, verified: read the paper →

Ground Truth. (2026, August 2). A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill. Ground Truth. https://groundtruth.day/news/a-284-billion-parameter-model-with-a-3-gigabyte-working-set.html

@misc{groundtruth:a-284-billion-parameter-model-with-a-3-gigabyte-working-set,
  title  = {A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/a-284-billion-parameter-model-with-a-3-gigabyte-working-set.html}
}

Topics: deepseek · local-ai · mixture-of-experts · inference · apple-silicon · open-source