News · 2026-09-01

A 104 GB model now runs on a 48 GB Mac by streaming experts off the SSD

A developer released slotstream on September 1, 2026, a single Swift binary that runs the 104 GB Qwen3.8-Flash-Next model on Macs that cannot hold it in memory, hitting about 12 tokens per second on a 48 GB M5 Pro while using roughly 32 GB. It works by keeping the model's small dense trunk resident and streaming the enormous routed-expert weights off the SSD into a fixed cache pool shared across all 48 layers. The Show HN post reached 150 points and 90 comments the same day.

Key facts

• Weights are 103.8 GB across 24 files at 4-bit; the project asks for roughly 110 GB of free disk and calls a 512 GB Mac the realistic minimum.
• On a 48 GB M5 Pro: about 12 tokens per second warm decode, about 2 seconds to start the engine (only the 3.8 GB trunk loads), 32 GB peak memory.
• Memory is auto-sized down to an 8.1 GB floor on an 8 GB Mac, where decode drops to roughly 3 tokens per second.
• Primary source: the slotstream repository; Hacker News discussion.

Mixture-of-experts models are the reason this trick is possible at all. In a dense model, every parameter participates in every token, so all of it has to be in fast memory. A mixture-of-experts model splits most of its parameters into many specialist sub-networks and routes each token through only a handful of them -- our explainer on mixture of experts covers the design. That means at any given moment, the overwhelming majority of the weights are idle. A 104 GB model might only need a few gigabytes of experts for the token it is currently producing.

The obvious move is to leave the idle experts on disk and fetch them as needed. The reason nobody gets this for free is a plumbing detail the README explains bluntly: Apple's MLX loader cannot materialize only a subset of a memory-mapped tensor. Ask for three experts out of a layer's 512 and you get all 512, which on a 48 GB Mac means the machine starts swapping before a single token comes out. slotstream sidesteps this by reading experts directly with pread into a fixed cache pool that every layer shares.

The analogy that fits is a professional kitchen with a small counter. You cannot fit the entire pantry on the counter, so you keep the things you touch constantly -- salt, oil, the knives -- permanently in reach, and you walk to the shelf for everything else. The trunk is the counter. The experts are the shelf. It works because you only need a few ingredients per dish, and it degrades exactly the way you would expect: the smaller your counter, the more walking you do. slotstream's own tier table makes this concrete, dropping from about 12 tokens per second at 48 GB to roughly 4 at 16 GB and 3 at 8 GB.

The honest limitation is not decode speed, it is the wait before decoding starts. The whole prompt gets processed before the first token appears, so an 8,000-token prompt takes about a minute on a 48 GB machine and over three minutes on a 16 GB one. Total context is capped at 32,768 tokens. Within a conversation you only pay that once -- follow-up turns prefill just the new text, and the project measures time-to-first-token staying flat at 6.0 seconds on the eighth turn versus 25.8 on the first. This is the prefill and decode split showing up in its purest form, and it is why why LLM inference is memory-bound is the single most useful thing to understand about local model performance.

There is a second gear. The model ships a draft head that predicts the token after next, and with speculative decoding enabled slotstream drafts a few tokens ahead and verifies them in one batched pass. The first draft is right 86 percent of the time, measured. But it only pays off when the expert cache is already near its best -- below about 26 GB of target memory the A/B test came out at 0.96x, slower -- so the feature defaults to off on smaller machines. The head also costs an extra 1.6 GB.

The engineering discipline around the download is worth noting, because this is where local-model tooling usually gets sloppy. All 24 files are checked against SHA-256 hashes compiled into the binary, so a truncated or corrupted download cannot reach the engine. Interrupted transfers resume. Releases are built by CI from the tagged commit with signed provenance, verifiable with gh attestation verify. And the README is candid that Hugging Face caps the transfer at roughly 36 to 57 MB/s regardless of how many connections you open -- a real install took 35 minutes on a fast link, about two hours and twenty minutes at 100 Mbps, and nine hours at 25.

For scale, the full-precision Qwen3.8-Flash-Next repository on Hugging Face runs to roughly 360 GB across 144 files, computed from the repository's own file listing. The 103.8 GB figure is the 4-bit conversion slotstream actually pulls -- a reminder that quantization is doing most of the heavy lifting before any streaming happens.

The honest caveat: only the 48 GB row of that performance table was measured on real hardware. The other tiers come from the project's own simulator, and the README says so, adding that smaller Macs also have slower SSDs than the curve assumes. There is also no head-to-head benchmark against llama.cpp in the repository, so claims about how this compares to the established local-inference stack remain untested. And the disk requirement bites before the memory one does: whatever RAM you have, you need a 512 GB drive. This lands in the same week Apple is pushing the Mac mini as an "always-on agentic" desktop at $899, and it is a sharper demonstration of what those machines can do than anything in Apple's own marketing -- though it also shows the ceiling, as we noted when Apple put 512 GB in a Mac Studio and bandwidth was still the wall.

Primary source, verified: read the paper →

Ground Truth. (2026, September 1). A 104 GB model now runs on a 48 GB Mac by streaming experts off the SSD. Ground Truth. https://groundtruth.day/news/slotstream-runs-a-104gb-model-on-a-48gb-mac.html

@misc{groundtruth:slotstream-runs-a-104gb-model-on-a-48gb-mac,
  title  = {A 104 GB model now runs on a 48 GB Mac by streaming experts off the SSD},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/news/slotstream-runs-a-104gb-model-on-a-48gb-mac.html}
}

Topics: local-inference · apple-silicon · mixture-of-experts · open-source · mlx · qwen · tools