News · 2026-08-02

Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token

A hand-written C engine generates text with Moonshot's Kimi K3, a 2.8-trillion-parameter model, using a peak of 8.24 gigabytes of RAM and no GPU at all. It produces roughly one token every 33 seconds. The result is real, the code is inspectable, and the framing that spread — an enormous model running on a modest machine — omits that the machine needs about 1.7 terabytes of fast local storage and is a dual-socket server.

Key facts

• The measurement: 8.24 GB peak RAM, CPU-only, at about one token every 33 seconds (roughly 0.03 tokens per second).
• The storage bill: the official checkpoint is about 1.56 TB, plus a roughly 109 GB repacked copy of the always-needed layers that the engine builds.
• The sparsity that makes it possible: only 16 of 896 expert modules fire for any given token.
• Primary source: the engine and its measurement log, with the model documented on Moonshot's official card.

The interesting part is what the engine does not do. It does not invent a new ultra-low-bit compression scheme. Kimi K3 already ships its routed expert weights in a four-bit format, and the engine simply reads those packed weights directly, along with the scaling factors that accompany them. Nothing is squeezed further; the released weights are used exactly as published.

What it exploits instead is sparsity. K3 is a mixture of experts, and for any given token a router selects just 16 of its 896 expert modules. The other 880 are irrelevant to that token, so there is no reason for them to occupy memory — they can stay on disk until they are needed. The engine also repacks the model's always-needed dense trunk layer by layer and streams it through a single rotating RAM buffer, so at any instant only one layer's worth of trunk is resident.

The result is a working set that fits in eight gigabytes and a storage requirement of roughly 1.7 terabytes. That is the trade, stated plainly: NVMe storage substituting for memory, at NVMe latency. It is why the number is 33 seconds per token rather than 33 tokens per second — a correction that had to propagate through the community, since the two read almost identically and differ by three orders of magnitude.

Three further qualifications matter. The run used a dual-socket EPYC workstation, so "one CPU" means one CPU-only inference process, and the four GPUs installed in that machine sat idle. Disk waits dominate the time, and results vary materially with NVMe performance; more RAM helps only modestly unless it pins a much larger share of the trunk. And long context is the hidden killer: the engine estimates roughly 2.4 gigabytes of conversation cache per thousand additional tokens, which means K3's million-token capability is entirely unusable at this memory setting.

The correctness checks are honest about their own limits. The author compared arithmetic and output probabilities against a reference implementation on a short synthetic prompt, and states explicitly that the incremental cache path — the code that runs during actual multi-token generation — was not exercised by that test.

K3's architecture is worth understanding here. It is a 93-layer multimodal mixture of experts activating about 104 billion parameters per token, built from 69 Kimi Delta Attention layers, 24 gated latent-attention layers and one dense layer. Only the latent-attention layers accumulate growing per-token cache, which is what makes long context tractable at all — but it does nothing to make the checkpoint small.

One correction to earlier coverage: the weights are downloadable. Moonshot's official card releases the full model under the Kimi K3 License with the files exposed in the repository, so the line that K3 remains unobtainable should be retired.

At the opposite end of the hardware spectrum, AMD validated a single K3 instance across eight MI355X GPUs using tensor parallelism, with enough high-bandwidth memory to keep the entire model resident. The stated functional check was a grade-school math benchmark at 16,000-token maximum length, and AMD explicitly declines to claim any throughput, latency or time-to-first-token figure. For what genuinely served performance looks like, vLLM reports 118 tokens per second per user on 16 NVIDIA GB300 NVL72 GPUs, rising to 370 with a speculative decoder — a different configuration entirely, and not comparable to AMD's result.

The lesson worth taking is that three things people routinely conflate are now visibly distinct: RAM capacity, downloadability, and usable local serving. K3 has been fitted into an eight-gigabyte envelope by treating 1.7 terabytes of NVMe as painfully slow memory. That is a genuine triumph of exact sparse-model systems work, and simultaneously proof that "runs on X gigabytes" has stopped being a meaningful headline. The same week produced a 284-billion-parameter model with a three-gigabyte working set making the identical point from the other direction.

The caveat is that this is a single developer's self-reported measurement, unreproduced by anyone else, with the incremental generation path unverified against a reference. The code is public, which makes it checkable — but nobody has checked it yet.

Primary source, verified: read the paper →

Ground Truth. (2026, August 2). Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token. Ground Truth. https://groundtruth.day/news/kimi-k3-runs-in-8-gigabytes-of-ram-at-33-seconds-per-token.html

@misc{groundtruth:kimi-k3-runs-in-8-gigabytes-of-ram-at-33-seconds-per-token,
  title  = {Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/kimi-k3-runs-in-8-gigabytes-of-ram-at-33-seconds-per-token.html}
}

Topics: kimi · local-ai · mixture-of-experts · inference · open-weight-models · systems