local-ai
The open world model ships inference and keeps the training code News
AlayaWorld released inference code and pretrained weights for an interactive world model with long-horizon memory, but the training code is still an unchecked box, the license is a community license, and running it requires a gated Google model plus a ByteDance depth model.
DeepSeek's new open model is 1.6 trillion parameters and runs 49 billion of them per token News
DeepSeek published DeepSeek-V4-Pro on Hugging Face with 1.6 trillion total parameters, 49 billion activated per token, and a one-million-token context window, making it the largest openly downloadable model of the current frontier wave.
China's biggest memory maker is booked through 2027 News
ChangXin Memory Technologies has reportedly sold out its DRAM output through the end of 2027 as PC brands rushed to secure supply, and consumer memory prices have stayed near their highs since.
llama.cpp shipped DSpark for DeepSeek V4 Flash, and almost everyone called it the wrong name News
llama.cpp release b10228 merged speculative-decoding support for DeepSeek V4 Flash, but the new 0731 checkpoint embeds DSpark and ships no MTP at all, so the widely repeated "MTP support landed" advice points users at a head their model does not contain.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
Offloading and streaming: running a model bigger than your memory Lesson
Offloading and streaming let a machine run a model far larger than its memory by keeping only the parts needed right now in fast memory and fetching the rest from system RAM or disk on demand, trading speed for capacity.
Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token News
A hand-written C engine generates text with Moonshot's 2.8-trillion-parameter Kimi K3 using a peak of 8.24 gigabytes of RAM and no GPU, by reading the model's four-bit experts directly off disk, at a rate of roughly one token every 33 seconds.
DeepSeek's low effort setting writes more than its high setting, because the dial is just a prompt News
A reproduction across both a local copy and DeepSeek's hosted API found V4 Flash consuming substantially more tokens on its low reasoning-effort setting than on high, and the released encoder explains why: low injects no instruction at all while high prepends a paragraph demanding exhaustive deliberation.
A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill News
An open-source engine called Mference runs DeepSeek V4 Flash on a 24 GB Mac with an effective memory footprint of about 3 gigabytes by streaming each token's experts off the SSD, but the checkpoint still occupies 90 to 98 gigabytes of disk and the test ran at a 4,000-token context.
A New Campaign Argues You Have a Right to Run AI on Your Own Computer News
A grassroots advocacy site, Right to Local Intelligence, is campaigning against proposed state laws it says could require a license just to download and run open AI models, framing local AI as the next personal computer.
Ollama nearly doubles Gemma's speed on Macs by guessing ahead News
A free local-AI tool now runs Google's Gemma model far faster on Apple computers using a trick where a small model drafts words and the big one checks them in bulk.
A model that rivals the frontier now squeezes onto a single high-end desktop News
Aggressive compression shrinks GLM 5.2 by more than 80 percent while keeping most of its accuracy, putting a near-frontier model within reach of local hardware.
llama.cpp b10228 Tool
The release that adds DeepSeek V4 Flash's embedded DSpark speculative-decoding head, plus a converter that can split the draft tensors into a separate GGUF. Gains are workload-dependent: roughly 2x decode on large multi-GPU setups, and a measured slowdown on a 24 GB card with CPU offload.
h3-studio Tool
Local front end for MiniMax H3 built on ComfyUI 0.30.0 or newer, with a VRAM meter, idle GPU release and a service setup so the video model hands the card back to other workloads. Its notes document H3 needing roughly 15.5 GB in the NVFP4 build, which is the practical ceiling on a 16 GB card.
bitnet.cpp Tool
Microsoft's official inference framework for 1.58-bit ternary language models, built on llama.cpp with optimized CPU and GPU kernels for running very heavily compressed models on ordinary hardware.
Unsloth Tool
Toolkit and documentation for running and fine-tuning large open models faster and on smaller hardware, including aggressive dynamic quantization recipes that shrink models like GLM 5.2 by 80-plus percent while keeping most of their accuracy. The practical on-ramp to running near-frontier models privately.
TurboQuant-MLX Tool
Quantization tooling for MLX with published size and speed measurements, including a 3-bit path that takes a 120-billion-parameter model from about 63GB down to 48GB on consumer Macs.
SwiftLM Tool
An MLX-based runtime for Apple silicon whose --stream-experts mode reads mixture-of-experts weights straight off an NVMe SSD, letting a machine run models several times larger than its RAM.
Ollama 0.31 Tool
Run open models on your own computer; the new version nearly doubles Gemma's speed on Apple Silicon using multi-token prediction, on by default.
Mference Tool
Runs DeepSeek V4 Flash on Apple silicon by keeping the shared core, attention and cache resident while streaming each token's routed experts off the SSD. Publishes an unusually honest memory budget: about 3 GB working set, 90 to 98 GB on disk, tested at a 4,000-token context on a 24 GB Mac, with no quality parity test yet.
Gemma-4 12B Coder (GGUF) Tool
A fine-tuned, locally-runnable version of Google's Gemma-4 model specialized for programming tasks, packaged in a format that runs efficiently on everyday consumer hardware.
GLM-5.2 Tool
A flagship openly-available language model with a very large context window for long documents and code. Free to download and run yourself, with compressed versions for more modest hardware.
GLM 5.2 (GGUF, runnable locally) Tool
Zhipu AI's open, MIT-licensed mixture-of-experts model with a roughly million-token context, now packaged as ready-to-run quantized files you can host on your own machine. Strong on agent and coding workflows; this week it beat Claude on a narrow security benchmark at a fraction of the cost.
DeepSeek-V4-Pro Tool
A downloadable 1.6-trillion-parameter mixture-of-experts model that activates 49 billion parameters per token, with a one-million-token context window under an MIT license. Serious server hardware required, but the weights are yours.
AlayaWorld Tool
Inference code and pretrained weights for an autoregressive world model with real-time camera control, prompt switching and long-horizon memory consistency, using an explicit 3D cache for spatial recall plus a compressed frame-history embedding. Training code is not included and the weights ship under a community license.