News · 2026-07-27

Kimi K3 is downloadable, but the floor to run it is eight datacenter GPUs

Kimi K3's weights are free to download and effectively impossible to run on a desktop. The practical floor for serving Moonshot's new 2.8-trillion-parameter model is a single eight-GPU node of NVIDIA B300s or AMD MI355Xs, according to launch-day writeups from vLLM, SGLang and AMD - and the popular claim that llama.cpp can already load a text-only version of it is not true.

Key facts

• 1.56 terabytes of natively quantized MXFP4 weights must stay resident across the GPU domain.
• Eight B300s or eight MI355Xs is the easiest supported deployment; 16 B200s also works, and 24 A100s is an experimental stretch.
• 27 July 2026: four separate runtime and hardware teams - Moonshot, vLLM, SGLang and AMD - published support material on release day.
• Primary source: vLLM's day-zero implementation post, which is unusually explicit about its own limits.

Here is the counterintuitive part. K3 routes each token to just 16 of its 896 expert networks, so the arithmetic per token is modest - about 104 billion parameters' worth. But routing is dynamic: any expert can be summoned at any moment, so all of them have to sit in fast memory, waiting. AMD's deployment analysis confirms this directly. Sparsity is a compute saving, not a storage escape hatch. Think of a hospital with 896 specialists on call: any given patient sees two or three of them, but you still have to pay all 896 to be in the building.

That gives a rough cost ladder, using Runpod's published per-GPU community rates on the day of release. Twenty-four A100s across three nodes barely clears the weights, lacks native FP4 support, and leaves nothing for runtime headroom - roughly 33 dollars an hour and not a recommended path. Sixteen H200s across two nodes fits, at about 70 dollars an hour, but every token pays a cross-node communication toll. Eight B300s in a single node - about 59 dollars an hour - is the first clean shape: enough memory for weights plus real headroom, and native FP4 arithmetic. The full Runpod price list is the source for those per-GPU rates; none of them is a quote for a fully networked cluster with storage.

The speed picture

vLLM reports roughly a hundred tokens per second for a single user on a much larger GB300 setup without speculation, rising to over three hundred with its K3-specific DSpark draft model. SGLang independently lands in the same territory: about a hundred tokens per second at batch size one, around four hundred with its speculator. If draft models are unfamiliar, our lesson on speculative decoding explains the trick.

SGLang's engineering account is the more revealing document. K3 mixes two incompatible kinds of state - mutable recurrent memory in its Kimi Delta Attention layers, and ordinary per-token key-value cache in its global attention layers. Prefix caching, paging, speculative decoding and prefill/decode disaggregation all had to be rebuilt so they could move both kinds safely. That is weeks of systems work, shipped on day zero.

AMD's contribution is the most honest of the three. It validated loading and a short correctness run on eight MI355X GPUs, and it explicitly declined to publish throughput, time-to-first-token or kernel efficiency numbers. To get even that far it skipped the vision tower, used an FP8 cache, capped context at 16,000 tokens and disabled prefix caching.

The llama.cpp claim does not hold up

Several posts this week suggested that text-only K3 support had landed in llama.cpp, and third-party GGUF files appeared on Hugging Face to match. The reality is messier. One community conversion shrinks the checkpoint from 1,453.8 GiB to 938.6 GiB - still a terabyte-class artifact - and its author states the result is "NOT loadable" in llama.cpp today. The conversion notes list what is missing: 896 experts exceeds llama.cpp's current limit, and Attention Residuals, Stable LatentMoE, the SiTU activation and a registered K3 architecture all still need implementing. The much-circulated llama.cpp pull request is a generic expert-streaming change validated on OLMoE and GLM-5.2, not merged K3 support.

Dropping the vision tower saves the small part. The roughly 1.5-terabyte language backbone is the problem, and text-only does not move the hardware floor.

The contrast that makes the point

While K3 needs a rack, a single-developer project called NInfer squeezes a 35-billion-parameter Qwen model onto one RTX 5090 with a 262,000-token context using an INT8 cache, reporting around 188 tokens per second at 250,000 tokens of context and roughly 584 to 695 tokens per second in its best speculative-decoding runs on structured workloads. Its methodology page is refreshingly specific about seeds, warm-up and its own limits: one active request, no continuous batching, no multi-GPU.

That is the real shape of the local-inference story right now, and it is not the one the headlines tell. Ruthless model-specific systems work can make a mid-sized model feel instantaneous on a one-dollar-an-hour rental. A frontier open model remains a multi-accelerator serving problem, because its active compute shrank far faster than its resident parameter mass. Our explainer on why inference runs out of memory bandwidth covers why that gap keeps widening, and our earlier piece on three ways the local inference floor fell tracks the other direction of travel.

The honest caveat

Every throughput number above comes from the team that wrote the runtime. None has been independently replicated, vLLM's recipe requires pre-release Docker dependencies, and AMD's numbers do not exist at all yet.

Primary source, verified: read the paper →

Ground Truth. (2026, July 27). Kimi K3 is downloadable, but the floor to run it is eight datacenter GPUs. Ground Truth. https://groundtruth.day/news/the-hardware-floor-for-kimi-k3-is-eight-datacenter-gpus.html

@misc{groundtruth:the-hardware-floor-for-kimi-k3-is-eight-datacenter-gpus,
  title  = {Kimi K3 is downloadable, but the floor to run it is eight datacenter GPUs},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/the-hardware-floor-for-kimi-k3-is-eight-datacenter-gpus.html}
}

Topics: inference · hardware · local-llm · vllm · sglang · open-weights · quantization