Learn · Intermediate
Offloading and streaming: running a model bigger than your memory
Offloading and streaming are the techniques that let a computer run a model far larger than its memory, by keeping only the pieces needed at this instant in fast memory and fetching everything else from system RAM or disk as it becomes relevant. The full model never has to fit anywhere at once. The price is speed: the machine now waits on data movement rather than on arithmetic, and how badly it waits depends entirely on the model's structure and your storage.
The problem starts with a mismatch. A large model's weights are measured in hundreds of gigabytes or terabytes. A good consumer GPU has 24 gigabytes of memory; a serious data-centre one has 80 to 192. The naive rule is that if the weights do not fit, you cannot run the model. That rule was true for a long time, and it is what makes headlines like "284-billion-parameter model in 3 gigabytes" sound impossible.
The way out is to notice that a model does not need all of itself simultaneously. A transformer runs layer by layer. When layer 12 is computing, layers 13 through 90 are idle, and their weights are just sitting there occupying space. So instead of loading everything, you load layer 12, use it, discard it, and load layer 13. The classic image is a chef with a small counter and a large pantry: you fetch one ingredient at a time, use it, put it back. The counter never needs to hold the whole pantry.
That basic idea has three levels, distinguished by where the not-currently-needed weights live.
Accelerator memory to system RAM. The fastest offload. System memory is roughly an order of magnitude slower than GPU memory but still fast, and the connection between them is wide. This is what DeepSpeed's ZeRO-Offload demonstrated for training, moving optimiser state and gradients to the CPU so a single GPU could train models that previously required a cluster.
System RAM to disk. ZeRO-Infinity extended the same idea to NVMe storage, treating solid-state drives as another tier of the memory hierarchy. This is what makes the truly extreme cases possible, and it is where the speed cost becomes severe: an SSD delivers gigabytes per second where accelerator memory delivers terabytes.
Streaming during generation. FlexGen showed how to schedule this movement for inference rather than training, overlapping computation with data transfer so the processor is not simply idle while it waits. The scheduling matters enormously. If you fetch layer 13 while computing layer 12, you hide much of the transfer cost. If you fetch it after, you pay all of it.
Here is where architecture becomes decisive, and why this technique has become so much more interesting recently. In a dense model, every parameter participates in every token, so every weight must be fetched for every token, and offloading is brutally expensive. In a mixture of experts, a router picks a small handful of specialist sub-networks per token — sixteen out of 896 in Moonshot's Kimi K3, for example. The other 880 are not slow to access; they are not accessed at all. That changes the arithmetic completely. Now the resident set is the shared trunk plus whichever few experts happen to be hot, and the disk only has to supply the rest on demand.
This is exactly what current projects exploit. Mference runs DeepSeek V4 Flash with about a three-gigabyte working set on a 24-gigabyte Mac by resident-loading the core and streaming routed experts off SSD — while the checkpoint still occupies 90 to 98 gigabytes of disk. A separate C engine runs Kimi K3 inside 8.24 gigabytes of RAM at one token every 33 seconds, backed by roughly 1.7 terabytes of storage.
Those two numbers — 8 gigabytes and 33 seconds — are the whole lesson in miniature. Offloading does not make a model smaller. It converts a capacity problem into a latency problem. Whether that trade is worth making depends on which one you actually have.
Three costs are easy to miss. First, the conversation cache grows independently of the weights, and it cannot be offloaded as cheaply because it is read on every step; one of the engines above estimates 2.4 gigabytes per thousand additional tokens, which is what makes long context the real killer rather than the weights. Second, expert routing is unpredictable — a token can route to a cold expert, so worst-case latency, not average, often determines whether a system feels usable. Third, throughput becomes memory-bandwidth-bound in the most literal sense: the storage device, not the processor, sets the pace, and a faster SSD buys more than a faster CPU.
The practical upshot for anyone running models locally: "does it fit" has stopped being the right question. The right questions are how much of the model is active per token, how fast your storage is, how long your contexts get, and whether your runtime overlaps transfers with computation. A model that technically runs at 0.03 tokens per second runs in the same sense that a car with no fuel pump rolls downhill.
ZeRO-Offload: Democratizing Billion-Scale Model Training
ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
Key questions
What is offloading in AI inference?
Why is offloading so much slower?
Why do mixture-of-experts models offload so well?
Cite this
APA
Ground Truth. (2026, August 2). Offloading and streaming: running a model bigger than your memory. Ground Truth. https://groundtruth.day/learn/offloading-and-streaming-weights.html
BibTeX
@misc{groundtruth:offloading-and-streaming-weights,
title = {Offloading and streaming: running a model bigger than your memory},
author = {{Ground Truth}},
year = {2026},
month = {aug},
url = {https://groundtruth.day/learn/offloading-and-streaming-weights.html}
}