News · 2026-08-04

A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type

A pull request to llama.cpp adds a runtime cache that watches which mixture-of-experts submodels a model actually calls, keeps the hottest ones in GPU memory, and computes the rest on the CPU. In the author's own tests on an 8GB card, decode speed on Qwen3.6-35B-A3B rose from 33.25 to 56.0 tokens per second at one quantisation and from 17.34 to 35.93 at another. PR #26563 is open and unmerged, CUDA-only, and off by default.

Key facts

• Enabled with -ehs / --expert-hot-s; CUDA-only for now; the PR page shows status Open with no merge and no maintainer approval in the visible thread.
• Author-reported gains: 33.25 to 56.0 tokens/second on Qwen3.6-35B-A3B at Q2_M, and 17.34 to 35.93 at Q5_K_P, both on 8GB of VRAM.
• Author-reported regressions on Qwen3.5-122B-A10B and Laguna-S-2.1; Gemma-4-26B-A4B excluded because both builds failed.
• Submitted to the ggml-org/llama.cpp repository, the engine behind most local LLM tooling.

The problem it attacks

A mixture-of-experts model splits its feed-forward layers into many specialists and routes each token through a handful of them. Qwen3.6-35B-A3B, the PR's main test subject, has 256 experts per layer and activates eight routed plus one shared per token, for about 3 billion active parameters out of 35 billion total.

For someone running this at home, the difficulty is not the arithmetic. It is geography. All 256 experts must live somewhere, and a consumer GPU cannot hold them. The usual answer is offloading: keep some weights in system RAM or on disk and move them when needed. Moving weights across PCIe is slow, and this site has covered the extreme version - a 26B model in 2GB by streaming experts off the SSD.

What the patch does differently

It stops treating every expert as equally likely. The PR tracks a heat map of expert usage during inference: per-layer counters that decay over time (default 0.999), ranked to pick the top S experts, with the ranking updated as tokens are generated. The hot store is populated after the first micro-batch, and a later commit adds periodic re-syncing to keep the resident set current as routing drifts.

At decode time the graph gains a MUL_MAT_ID_COLD path that skips hot experts on the CPU side, plus a hook combining hot-lookup remapping, the cold operation, scale handling and the add step. The key consequence: a cold expert selection does not trigger a weight transfer. The hot tier runs on GPU, the cold tier runs on CPU, and nothing shuttles across the bus mid-token.

Thrashing is the obvious failure mode, and the author guards against it with hysteresis - a cold expert must score at least 1.3 times the resident one before it takes its slot. Think of a small shelf beside your desk: you keep the books you actually reach for, and you do not swap one out just because you glanced at another once.

One detail is widely misread. The "8GB" in the benchmark is the card, not the cache. The autofit path measures free VRAM after the rest of the model is placed, converts the leftover bytes into a number of expert slots, and then forces all experts to CPU so the hot-store copy reads from host pointers. The PR reports 126 slots for the Q2_M run and 43 for Q5_K_P, and does not print cache size in bytes.

Why it matters

Local MoE inference has been stuck on a framing problem. The question everyone asks is "does the model fit," and the answer for anything interesting is no. The question this patch asks instead is "which slice is being used right now, and can it live where it runs fastest." Routing in a trained MoE is not uniform - some experts are called far more often than others - so a cache that learns the distribution at runtime is exploiting real structure rather than guessing.

It also arrives the same week that SK hynix and Sandisk published a memory-tier spec aimed at exactly this problem in datacentre silicon. Same bottleneck, two very different timescales: one is a 2027 accelerator architecture, the other is a flag you could compile tonight.

The honest caveat

This is one contributor's benchmark of one branch, and the reception is skeptical rather than celebratory. GitHub's bot flagged the PR as large and lacking prior discussion. A reviewer noted it looks similar to an existing RFC discussion; the author replied that the approach differs and offered to maintain the code and make it more idiomatic. No maintainer has endorsed it in the visible thread.

The benchmark disclosure is thin, too: no GPU model, CPU, RAM, PCIe configuration, prompt or warm-up procedure. And the regressions are as informative as the wins - two models got slower, one commenter saw a slowdown on older hardware, and one test was thrown out because neither build ran. A runtime cache that learns is a genuinely good idea whose value depends entirely on how concentrated the routing distribution happens to be for your model. Sometimes it is not.

Primary source, verified: read the paper →

Ground Truth. (2026, August 4). A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type. Ground Truth. https://groundtruth.day/news/a-llama-cpp-patch-learns-which-experts-to-keep-in-vram-while-you-type.html

@misc{groundtruth:a-llama-cpp-patch-learns-which-experts-to-keep-in-vram-while-you-type,
  title  = {A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/a-llama-cpp-patch-learns-which-experts-to-keep-in-vram-while-you-type.html}
}

Topics: llama-cpp · moe · local-llm · inference · open-source · cuda · offloading