local-inference
Compile by Training turns a language specification into a reusable local neural function News
A new EMNLP demonstration system uses teacher-generated examples to train a compact task-specific adapter from a natural-language specification, reporting 83.6% semantic accuracy on a difficult subset where a fast compiler achieved 22.4% mean LEM.
A 104 GB model now runs on a 48 GB Mac by streaming experts off the SSD News
slotstream, a single Swift binary released as a Show HN on September 1, 2026, runs the 104 GB Qwen3.8-Flash-Next mixture-of-experts model on Macs with a fraction of that memory by keeping a small trunk resident and reading expert weights off the SSD on demand -- about 12 tokens per second on a 48 GB machine.
llama.cpp merged Qwen's new architecture and a 97-gigabyte lookup table News
Support for Qwen3.8-Flash-Next landed in llama.cpp on August 27, adding a sparse-attention graph, vision, three quantizer fixes and machinery to stream a 97.7 GiB n-gram table that never has to sit on the GPU.
DRAM contract prices nearly doubled in a single quarter News
Conventional memory contract prices rose roughly 93% to 98% quarter over quarter in early 2026 and are forecast to climb another 58% to 63%, as suppliers divert capacity to AI servers -- repricing the exact component local AI depends on.
Apple's Mac Studio now holds 512GB of unified memory, which solves capacity and leaves speed exactly where it was News
The M5 Ultra Mac Studio configures to 512GB of unified memory at 1.2TB/s, enough to load almost any open-weight model in existence, but its memory bandwidth still sets a hard ceiling on how fast those models can generate text.
A 22-gigabyte local coder matched Opus on a 25-problem slice News
TielCoder, a 4-bit re-quantization of the Ornith-1.5 mixture-of-experts model, fits in 22.4 GB and fixed 12 of 25 recent real-world coding issues in its maintainer's tests, the same count as Claude Opus 4.6 at medium effort.
A llama.cpp fork is reviving $200 AMD cards nobody else supports News
A specialist fork of llama.cpp ships hand-written kernels for AMD's decade-old GFX906 architecture, making cheap used MI50 and Radeon VII cards usable for local inference, and upstream maintainers are now discussing porting the work back.
Qwen3.8-27B's compression data has a blind spot where its speed-up head lives News
The importance matrix used to compress Qwen3.8-27B contains no entries for block 64, the model's multi-token-prediction head, because that head never activates during the standard calibration run, which is where the most aggressive one-bit builds start to break.
llama.cpp tagged v0.1.0 after ten thousand build numbers News
The most widely used local AI inference engine published its first semantic version tag on August 17, 2026, after years of releasing by build number, while a pending contribution adds a speculative-decoding mode that adjusts its own draft depth on the fly.
Two-bit models now run on every major llama.cpp backend News
The official 2-bit weight format merged into llama.cpp now works on mainline CPU, Metal, Vulkan, and CUDA after the CUDA path landed upstream, completing a backend sweep that started as an ARM-only contribution.
llama.cpp ships the fix that lets DeepSeek V4 Flash call tools mid-thought News
DeepSeek's new open-weight model shipped without a standard chat template, and local agents kept dying when it emitted a tool call inside its reasoning; llama.cpp release b10217 and KoboldCpp v1.118 both landed fixes on 1 August.
A robot control model now runs 32 times a second on a gaming GPU, in under a gigabyte News
TurboVLA reaches real-time robot control at 32 Hz using 0.9GB of memory on a consumer RTX 4090, by removing the large language model from the control loop entirely rather than compressing it.
A 26-billion-parameter model runs in 2GB of RAM by streaming experts off the SSD News
TurboFieldfare, an open-source Swift and Metal runtime, runs Gemma 4's 26-billion-parameter model on an 8GB MacBook Air by keeping only a 1.35GB core in memory and pulling each token's experts from disk as it needs them.
DeepSeek V4 Flash hits 32 tokens a second on one desktop News
A published benchmark shows DeepSeek's 284-billion-parameter V4 Flash generating 32 tokens per second entirely on one AMD Strix Halo machine, using aggressive quantization, speculative decoding and reduced expert routing.
llama.cpp Merges MiniMax M3's Sparse Attention, Because Running It Dense Gives Wrong Answers News
Support for MiniMax M3's block-sparse attention landed in llama.cpp today, and the contributor is explicit that it is not a speed optimisation: the model was trained sparse, so running it dense produces degraded output.
llama.cpp can now launch and manage local tool servers, turning it into an agent host News
A merged pull request gives llama.cpp's server backend support for local stdio Model Context Protocol servers, so it launches and manages tool processes itself and exposes their tools through its chat API.
A complete text-to-speech system now fits in 9.4 million parameters News
Inflect-Micro-v2 packs an entire English speech synthesis stack, including the waveform decoder, into 9,356,513 parameters that run locally with no external vocoder or API.
Three separate tricks dropped the hardware floor for local AI in one day News
A 26-billion-parameter Gemma model ran on an iPhone by streaming expert weights from storage, AMD shipped a sparse mixture-of-experts model activating 2.8 billion parameters per token, and a llama.cpp fork began saving conversation caches to disk - three unrelated attacks on three different bottlenecks.
Poolside's Laguna S 2.1 shipped with a broken chat template - and the fixes explain the reviews News
Days after releasing its open coding model, Poolside has been repairing it in public: the base chat template shipped with reasoning disabled by default and a 32,768-token generation cap, and its own quantised builds needed re-releases to fix tool calls and thinking.
Poolside's Laguna S 2.1 is a small open coding agent with big benchmark claims News
Poolside released Laguna S 2.1, a public-weight coding model with an unusually low 8 billion active parameters that runs locally on a single high-end machine, but its claims of beating DeepSeek V4 Pro come from the company's own benchmark table and one early hands-on tester found it fabricates facts when evidence runs out.
Nanbeige4.2-3B reuses one 22-layer stack twice to punch above its size News
A Chinese lab released Nanbeige4.2-3B, a small open-weight model that runs its 22 transformer layers twice in sequence to get 44 layers of depth from one set of weights, posting benchmark numbers rivaling models three times its size, though the results are vendor-reported and a widely repeated 'beats 4x its size' claim does not survive clean accounting.
speech-to-speech Tool
Hugging Face's modular local voice-agent pipeline - voice detection, speech recognition, a language model and text-to-speech chained together, with an OpenAI Realtime-compatible websocket so existing clients can point at it.
slotstream Tool
A single Swift binary that runs the 104 GB Qwen3.8-Flash-Next mixture-of-experts model on Apple Silicon Macs with far less memory, by streaming expert weights off the SSD. Speaks the Ollama and OpenAI chat APIs, so existing tools work unchanged. About 12 tokens per second on a 48 GB Mac; needs roughly 110 GB of free disk.
llama.cpp-gfx906 Tool
A llama.cpp fork with hand-written kernels for AMD's GFX906 architecture, making used Instinct MI50, MI60, and Radeon VII cards usable for local inference. Ships custom flash-attention, RoPE, and matrix-multiply paths plus overclocking and power-scaling scripts.
llama.cpp v0.1.0 Tool
The engine behind most local AI setups published its first semantic-looking version tag on August 17, 2026, pinned to commit 7c35571. Useful mainly to packagers and anyone who needs a version string a dependency resolver understands; the project still makes no API-stability promise.
llama.cpp b10217 Tool
The 1 August build adds support for DeepSeek V4 Flash emitting tool calls inside its reasoning block, which is what was silently killing local agent runs against the new model. If you are running DS4 locally with tools, this is the build you need.
llama.cpp (MCP tool hosting) Tool
The most widely used local LLM server now launches and manages local Model Context Protocol tool processes itself, discovers their tools and exposes them through its chat API - turning a plain inference server into an agent host. Off by default; needs a tool-capable chat template.
WASTE Tool
A C inference engine that streams only the experts a mixture-of-experts model actually activates directly off NVMe, using spare RAM as an expert cache. Its author reports running the full 2.78-trillion-parameter Kimi K3 on a 64 GB laptop at about half a token per second. An existence proof, not a chat app.
Unsloth Kimi-K3-GGUF Tool
Converted local-inference builds of Moonshot's Kimi K3: a 1.51 TB four-bit UD-Q4_K_XL file, a 1.56 TB eight-bit build, and BF16/F16/F32 multimodal projector files that preserve an image-input path. Datacenter-scale hardware still required.
TurboFieldfare Tool
A Swift and Metal runtime that runs Gemma 4's 26B model on an 8GB MacBook Air by keeping a 1.35GB core resident and streaming the rest of the experts off the SSD. Ships as a Mac app, a CLI and an OpenAI-compatible local server.
TielCoder 35B-A3B GGUF Tool
4-bit dynamic re-quantization of Ornith-1.5-35B-A3B for llama.cpp. The benchmarked 22.4 GB tier fits a 24 GB card and fixed 12 of 25 live software issues in the maintainer's tests. Includes a vision projector for reading screenshots and stack traces.
Ternary Bonsai models Tool
A family of 1.7B, 4B and 8B models built for extreme quantization, shipped in the official group-64 two-bit format that mainline llama.cpp reads. Useful if you want to see what 2-bit inference feels like without converting anything yourself.
Qwen3.8-Flash-Next GGUF quants Tool
Unsloth's quantized builds of Qwen's newest architecture, in eleven sizes from roughly 72.5 GB at the smallest to about 354 GB at full precision. No official VRAM figure is published; community reports run a 4-bit build on a 16 GB card with around 100 GB of combined system memory.
Qwen3.8-Flash-Next GGUF (Unsloth) Tool
Unsloth's quantized GGUF conversions of Qwen3.8-Flash-Next, including a 2-bit UD-Q2_K_XL build at roughly 78.9 GB across three shards -- about a fifth of the official bf16 repository. The model card carries working setup instructions for llama.cpp, vLLM and Ollama.
Qwen3.8-27B GGUF builds Tool
Ready-to-run compressed builds of Alibaba's 27-billion-parameter multimodal Qwen3.8, covering the full ladder from eight-bit down to one-bit. Community testing points to the six-bit build, around 22 gigabytes, as the conservative floor for serious agentic coding, with three-bit still usable and one-bit rebuilds degrading sharply because the calibration file has no data for the model's multi-token-prediction head.
Poolside Laguna S 2.1 (GGUF) Tool
Open-weight 118B mixture-of-experts coding agent activating about 8B parameters per token, under the permissive OpenMDW-1.1 licence, in GGUF plus FP8, NVFP4 and INT4 builds. Use the current re-released Q4/Q8 files - the initial ones shipped with a broken chat template.
Poolside Laguna S 2.1 Tool
A public-weight, 118B-total mixture-of-experts coding model with only ~8B active parameters that runs locally on a single 128GB machine via a 75GB Q4 GGUF, built for long-horizon agentic software work under the permissive OpenMDW-1.1 license.
Ornith-1.5 Tool
Three open coding and agentic models -- 397B and 35B mixture-of-experts plus a 9B dense model with a quantized Mobile build for phones. The 9B is single-GPU at roughly 19 GB with a 262,144-token context and OpenAI-compatible tool calling; the flagship matches Claude Opus 4.8 on terminal-coding benchmarks.
Noema Overfit Tool
Repackages compatible mixture-of-experts model files so shared weights stay resident in memory while expert weights stream from local storage on demand, letting phones load models far larger than their RAM. Experimental, and slower than a smaller fully-resident model on short prompts.
Nanbeige4.2-3B Tool
An Apache-2.0 4B model that reuses one 22-layer transformer stack twice for 44 layers of depth from a single set of weights, shipping BF16 weights with SGLang, vLLM, llama.cpp, and Ollama paths for local use.
KoboldCpp v1.118 Tool
Single-binary local model server that shipped its own fix for multi-turn DeepSeek V4 Flash prompt-processing problems on the same day as the llama.cpp fix. Useful if you want a working DS4 setup without building anything.
Inkling-Small GGUF Tool
Quantized builds of Thinking Machines' newly released 276B/12B multimodal open-weight model, packaged for llama.cpp, LM Studio and Ollama so you do not have to download the 532GB original.
Hugging Face speech-to-speech Tool
Local voice-activity detection to speech recognition to language model to text-to-speech pipeline, threaded through queues and exposed as an OpenAI Realtime-compatible server so existing clients can point at it unchanged.
Free Claude Code Tool
MIT-licensed local proxy that lets Claude Code, Codex, OpenCode and other coding agents run against roughly 50 different providers, preserving Anthropic's wire protocol so the client never notices. Routes each internal model tier to a different upstream.
DeepSeek-V4-Flash-0731 GGUF (Unsloth) Tool
Community quantizations of the new MIT-licensed DeepSeek weights in GGUF form, running from roughly 83GB at aggressive low precision to about 162GB at 8-bit. Usable on high-memory workstations and multi-GPU rigs, not on a laptop.
CachyLLama Tool
MIT-licensed llama.cpp fork that saves conversation and system-prompt caches to SSD and restores them after a restart, so local agents stop reprocessing the same prompt prefix every turn. Its own benchmark reports long repeated agent prefixes going from minutes cold to about a second warm.
AIRI Tool
Self-hosted embodied assistant with a Live2D or VRM character, voice, persistent memory, local inference support, and game and chat integrations. A vertical application rather than a general agent framework.