News · 2026-10-18

Xolotl streams a 152 GiB model through one consumer graphics card

Xolotl’s developer reports running the full 552-billion-parameter DeepSeek-V4.1-Flash model on a computer with one 24 GB RTX 4090 by spreading its 152 GiB weight file across graphics memory, system memory, and solid-state storage. The project reports about 27 output tokens per second on short coding prompts. Its achievement is coordinated memory use, with performance and answer quality still supported by developer measurements rather than independent replication.

Key facts

• The documented compressed weight file occupies 152 GiB on disk.
• The reported machine has a 24 GB RTX 4090, 90 GB of system memory, a Ryzen 7 5700X processor, and fast solid-state storage.
• The benchmark notes report 26.8 output tokens per second on short answer-writing prompts; concurrency is unspecified.
• The primary sources are the Xolotl repository and its benchmark notes.

The model is larger than the graphics card

The phrase “on one 4090” can suggest that a huge model somehow fits inside the card. Xolotl does something more interesting and more conditional: it makes the rest of the computer part of the inference engine.

DeepSeek-V4.1-Flash uses a mixture of experts, meaning that different parts of the network handle different tokens. The complete model contains many experts, but its router selects only some of them for each step. Xolotl exploits that selective use by arranging experts across three memory tiers.

Frequently needed experts stay in graphics memory. A larger collection sits in pinned system memory, prepared for transfers to the graphics card. The remainder lives on fast solid-state storage and is fetched as needed. The developer also describes prefetching work one layer ahead and moving frequently used experts into the system-memory arena between prompts. The repository’s mechanism guide explains the arrangement.

Think of a cook with a small countertop, a pantry, and a warehouse. A recipe can draw on the warehouse’s full inventory without putting every ingredient on the counter simultaneously. The result depends on anticipating what comes next and avoiding repeated trips for ingredients that should have stayed nearby.

Disk space and working memory answer different questions

The 152 GiB figure describes the project’s compressed weights on disk. It is not a graphics-memory requirement. The reported working configuration uses a 24 GB graphics card alongside 90 GB of system memory and storage; the sources do not establish a universal minimum graphics-memory requirement for every context length or workload.

Other components also affect memory and speed. Xolotl documents mixed expert compression, disk-backed phrase memory, speculative decoding, and a four-bit attention cache. These choices change how much information is kept in each tier and how often computation or transfers can be avoided. Readers considering a download should therefore reproduce the documented software and hardware arrangement, rather than equating the checkpoint size with a suitable graphics card.

Our lesson on offloading and streaming weights explains the general tradeoff: capacity increases, but data movement becomes part of the critical path.

The speed result needs its workload attached

The benchmark notes describe real-server evaluation rather than only a synthetic throughput test. They cover 21 coding problems, ten tool-call tasks, and three agentic bug-fix loops, with thinking enabled and a 32,000-token thinking budget. The headline 26.8-token rate comes from short answer-writing prompts, and longer-context runs are slower.

The document labels its timings “per-request.” It does not state concurrency, so the number does not establish either a single-stream measurement or the capacity to serve many simultaneous users. Nor does a token-per-second figure reveal how many reasoning tokens the system needs to finish a useful task.

That last distinction is especially important here. In the developer’s Reddit discussion, the author said his compressed version needed roughly ten times as many thinking tokens to reach the same outcome as the hosted service in his own comparison. That is an anecdote, not a controlled quality study. It is nevertheless a stronger warning than a generic statement that compression may reduce accuracy.

The useful result is a reproducible systems claim

Xolotl offers a public implementation and a harness other developers can attempt to reproduce. That makes it more useful than an isolated screenshot, while leaving the measurements explicitly self-reported.

The opportunity is access to a large model using a consumer graphics card plus substantial supporting memory and storage. The caveat is that output speed, reasoning efficiency, and final answer quality must be measured together. A system that prints quickly but thinks much longer may still finish the job more slowly.

Primary source, verified: read the paper →

Ground Truth. (2026, October 18). Xolotl streams a 152 GiB model through one consumer graphics card. Ground Truth. https://groundtruth.day/news/xolotl-streams-152-gib-model-across-4090-ram-storage.html

@misc{groundtruth:xolotl-streams-152-gib-model-across-4090-ram-storage,
  title  = {Xolotl streams a 152 GiB model through one consumer graphics card},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/xolotl-streams-152-gib-model-across-4090-ram-storage.html}
}

Topics: local-ai · inference · open-source · mixture-of-experts · hardware