offloading
A 2.8-trillion-parameter model runs on a laptop by streaming weights off four SSDs News
A project called Deltafin runs the full uncompressed Kimi K3 model — 2.8 trillion parameters, with 1.45 TB of expert weights — on a single MacBook Pro by streaming experts from four external SSDs on demand, sustaining almost exactly one token per second.
A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type News
An open pull request to llama.cpp tracks which mixture-of-experts submodels get used most during inference and promotes them to GPU memory on the fly, roughly doubling decode speed on an 8GB card in the author's own tests - while slowing other models down.
Offloading and streaming: running a model bigger than your memory Lesson
Offloading and streaming let a machine run a model far larger than its memory by keeping only the parts needed right now in fast memory and fetching the rest from system RAM or disk on demand, trading speed for capacity.
Edge0-35B-A3B-preview Tool
A preview mixture-of-experts model released 8 September under Apache 2.0, with a runtime that streams experts from the SSD so the model can run in a small memory footprint on a laptop. The weights are a 19.7 GB download; speed figures on the card are the developers' own.
Deltafin Tool
A Rust engine that runs the full uncompressed 2.8-trillion-parameter Kimi K3 on a single MacBook Pro by streaming expert weights from external SSDs, sustaining about one token per second with nothing quantized or pruned. A demonstration of the ceiling on local inference rather than a daily driver.