offloading
Lean-QwenFlash reports a 125B model at 20 tokens a second with 16 GB of GPU memory News
Lean-QwenFlash streams model experts from solid-state storage and reports 20 tokens a second at a 262,000-token context, with quality retention still unmeasured.
A 2.8-trillion-parameter model runs on a laptop by streaming weights off four SSDs News
A project called Deltafin runs the full uncompressed Kimi K3 model — 2.8 trillion parameters, with 1.45 TB of expert weights — on a single MacBook Pro by streaming experts from four external SSDs on demand, sustaining almost exactly one token per second.
A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type News
An open pull request to llama.cpp tracks which mixture-of-experts submodels get used most during inference and promotes them to GPU memory on the fly, roughly doubling decode speed on an 8GB card in the author's own tests - while slowing other models down.
Offloading and streaming: running a model bigger than your memory Lesson
Offloading and streaming let a machine run a model far larger than its memory by keeping only the parts needed right now in fast memory and fetching the rest from system RAM or disk on demand, trading speed for capacity.
TensorSharp Tool
Public .NET inference engine for local model execution, including documented SSD-backed workflows; the specific 16 GB laptop demonstration remains unverified.
Lean-QwenFlash Tool
Linux/NVIDIA runtime that streams Qwen expert weights from storage; its documented setup uses an approximately 85 GB download, a packed store, and a demonstrated 16 GB of GPU memory, with quality retention unmeasured.
Edge0-35B-A3B-preview Tool
A preview mixture-of-experts model released 8 September under Apache 2.0, with a runtime that streams experts from the SSD so the model can run in a small memory footprint on a laptop. The weights are a 19.7 GB download; speed figures on the card are the developers' own.
Deltafin Tool
A Rust engine that runs the full uncompressed 2.8-trillion-parameter Kimi K3 on a single MacBook Pro by streaming expert weights from external SSDs, sustaining about one token per second with nothing quantized or pruned. A demonstration of the ceiling on local inference rather than a daily driver.