local-ai
Tether releases BotPsy, a small model for tool-calling applications News
BotPsy is a downloadable 0.8-billion-parameter model for developers, with a 529 MB Q4 weight file.
Claris previews Harbor for running local AI across Macs News
Claris has listed Harbor as a private preview for local model serving and multi-Mac clustering.
Altman argues for pooled compute and reveals the uncertainty behind agent-led math News
Sam Altman defended cloud memory pooling and described roughly $10 million for OpenAI’s Navier–Stokes effort, without an audited cost breakdown.
Nvidia releases multilingual Nemotron speech transcription model News
Nvidia’s Nemotron 3.5 Transcribe supports offline transcription in 40 languages, while a circulating dramatic-pruning claim remains unverified.
Liquid releases six LFM3 specialists, including models that decide without chatting News
Liquid AI’s LFM3 family pairs small local text, vision, and speech models with typed decision outputs under a revenue-limited license.
llama.cpp advisory flags a crafted Pocket-TTS model-file risk News
A llama.cpp advisory identifies a local out-of-bounds write in Pocket-TTS model loading, with update guidance that should not be mistaken for proof of a complete root-cause fix.
Nanbeige releases a 33B teacher model for a planned smaller-agent pipeline News
Nanbeige4.3-33B offers Apache-2.0 weights and lab-reported agent benchmarks, while its proposed smaller student models remain future work.
RTX compute unlocks reach Ada and Turing, with gains narrower than the 2× headlines News
Separate open-source projects report faster selected tensor operations on named RTX 20, 30 and 40 cards, without establishing a universal doubling of chatbot speed.
Lean-QwenFlash reports a 125B model at 20 tokens a second with 16 GB of GPU memory News
Lean-QwenFlash streams model experts from solid-state storage and reports 20 tokens a second at a 262,000-token context, with quality retention still unmeasured.
ISTA’s two-bit DeepSeek quant matches three reported tests in a 249 GB package News
ISTA-DASLab’s compact DeepSeek release matches its base model on three reported benchmarks, while requiring a 248.84 GB download and substantial memory outside the GPU.
Xolotl streams a 152 GiB model through one consumer graphics card News
Xolotl reports running DeepSeek-V4.1-Flash using a 24 GB RTX 4090, system memory, and a 152 GiB weight file streamed from storage.
Apple ran Muse Spark Open at 36.73 tokens a second across four Mac Studios News
Apple reports 36.73 tokens per second for Muse Spark Open on four M5 Ultra Mac Studios, with slower two-machine configurations documented separately.
Saluki offers a Qwen-derived coding and tool model in a 7.89 GB file News
Saluki’s publisher offers a 7.89 GB Qwen-derived model and reports task-dependent quality retention, with limited benchmark comparability and unstated running-memory requirements.
Strata users report roughly 100 tokens a second on a gaming GPU—with substantial RAM News
Strata users report roughly 100 generated tokens per second on an RTX 4090 by combining compressed weights, expert caching and substantial system RAM.
Redis creator antirez's ds4 runs DeepSeek-class models on a 128GB Mac, and it is back in the spotlight News
ds4, a small C inference engine by Redis creator Salvatore Sanfilippo, runs large open models such as DeepSeek V4 Flash on 96-128GB machines; a community guide brought the May project back to Hacker News on October 2.
NVIDIA's cheaper DGX Spark halves the memory and starts at $4,999 News
NVIDIA announced a 64GB DGX Spark desktop AI system from six PC makers starting at $4,999 on October 23, while its own 128GB model is listed at $6,950.
llama.cpp merges Qwen multi-token decoding with a reported 55% speed gain News
llama.cpp merged multi-token prediction support for Qwen3.8-Flash-Next, with a small author-run test reporting about 55% higher decoded throughput.
Micron forecasts tighter memory and storage supply through 2028 News
Micron expects memory and storage supply-demand conditions to tighten in 2027 and 2028 despite expanding factory investment.
llama.cpp adds GLM-5.3-Flash support, bringing a 328 GB model into the local-runtime ecosystem News
llama.cpp merged support for Z.ai’s MIT-licensed GLM-5.3-Flash on September 30, but the official model repository still occupies 328 GB.
Jeff releases a small open model for fast typed decisions on local hardware News
Jeff is an open 0.8B decision-model project that returns calibrated option choices instead of prose, illustrating where a small local model can be useful without claiming frontier reasoning ability.
Kev brings Jev-like local decision models to Qwen, while audits challenge universal calibration claims News
Kev released open local models for typed probability decisions, as published evaluations and an out-of-distribution audit show that confidence calibration varies sharply by task.
Apple's M5 Ultra makes large local AI practical by capacity, not by beating an RTX 5090 News
A 256 GB M5 Ultra Mac Studio ran large sparse models and long contexts locally in a detailed review, but an RTX 5090 remained faster whenever the tested workload fit in its VRAM.
Bonsai 2 ships a 27B local model in a 5.95 GB download News
Prism's ternary Bonsai 2 compresses its smallest 27B language-weight pack to 5.95 GB, but the headline is a disk-size claim rather than a full runtime or VRAM requirement.
A 753 billion parameter model ran on a single workstation GPU News
A serving system called FreeToken reports running frontier-scale sparse models on ordinary personal hardware, including a 753 billion parameter model on one workstation GPU and a 35 billion parameter model on a laptop with 8 gigabytes of video memory.
The open world model ships inference and keeps the training code News
AlayaWorld released inference code and pretrained weights for an interactive world model with long-horizon memory, but the training code is still an unchecked box, the license is a community license, and running it requires a gated Google model plus a ByteDance depth model.
DeepSeek's new open model is 1.6 trillion parameters and runs 49 billion of them per token News
DeepSeek published DeepSeek-V4-Pro on Hugging Face with 1.6 trillion total parameters, 49 billion activated per token, and a one-million-token context window, making it the largest openly downloadable model of the current frontier wave.
China's biggest memory maker is booked through 2027 News
ChangXin Memory Technologies has reportedly sold out its DRAM output through the end of 2027 as PC brands rushed to secure supply, and consumer memory prices have stayed near their highs since.
llama.cpp shipped DSpark for DeepSeek V4 Flash, and almost everyone called it the wrong name News
llama.cpp release b10228 merged speculative-decoding support for DeepSeek V4 Flash, but the new 0731 checkpoint embeds DSpark and ships no MTP at all, so the widely repeated "MTP support landed" advice points users at a head their model does not contain.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
Offloading and streaming: running a model bigger than your memory Lesson
Offloading and streaming let a machine run a model far larger than its memory by keeping only the parts needed right now in fast memory and fetching the rest from system RAM or disk on demand, trading speed for capacity.
Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token News
A hand-written C engine generates text with Moonshot's 2.8-trillion-parameter Kimi K3 using a peak of 8.24 gigabytes of RAM and no GPU, by reading the model's four-bit experts directly off disk, at a rate of roughly one token every 33 seconds.
DeepSeek's low effort setting writes more than its high setting, because the dial is just a prompt News
A reproduction across both a local copy and DeepSeek's hosted API found V4 Flash consuming substantially more tokens on its low reasoning-effort setting than on high, and the released encoder explains why: low injects no instruction at all while high prepends a paragraph demanding exhaustive deliberation.
A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill News
An open-source engine called Mference runs DeepSeek V4 Flash on a 24 GB Mac with an effective memory footprint of about 3 gigabytes by streaming each token's experts off the SSD, but the checkpoint still occupies 90 to 98 gigabytes of disk and the test ran at a 4,000-token context.
A New Campaign Argues You Have a Right to Run AI on Your Own Computer News
A grassroots advocacy site, Right to Local Intelligence, is campaigning against proposed state laws it says could require a license just to download and run open AI models, framing local AI as the next personal computer.
Ollama nearly doubles Gemma's speed on Macs by guessing ahead News
A free local-AI tool now runs Google's Gemma model far faster on Apple computers using a trick where a small model drafts words and the big one checks them in bulk.
A model that rivals the frontier now squeezes onto a single high-end desktop News
Aggressive compression shrinks GLM 5.2 by more than 80 percent while keeping most of its accuracy, putting a near-frontier model within reach of local hardware.
ruNNtime Tool
Public WebGPU inference framework documented for browsers, Electron and React Native; official support for Liquid d1 was not established.
llama.cpp b10228 Tool
The release that adds DeepSeek V4 Flash's embedded DSpark speculative-decoding head, plus a converter that can split the draft tensors into a separate GGUF. Gains are workload-dependent: roughly 2x decode on large multi-GPU setups, and a measured slowdown on a 24 GB card with CPU offload.
h3-studio Tool
Local front end for MiniMax H3 built on ComfyUI 0.30.0 or newer, with a VRAM meter, idle GPU release and a service setup so the video model hands the card back to other workloads. Its notes document H3 needing roughly 15.5 GB in the NVFP4 build, which is the practical ceiling on a 16 GB card.
ds4 (DwarfStar 4) Tool
A small MIT-licensed C engine by Redis creator antirez that runs a short list of tested large open models, including DeepSeek V4 Flash, GLM 5.3 and Qwen3.8 Flash Next, on Metal, CUDA and ROCm; upstream recommends 96-128GB of memory for DeepSeek V4 Flash at 2 bits.
bitnet.cpp Tool
Microsoft's official inference framework for 1.58-bit ternary language models, built on llama.cpp with optimized CPU and GPU kernels for running very heavily compressed models on ordinary hardware.
audio.cpp Tool
The September 30 v0.9.0 README reports a shared local audio runtime covering more than 100 models and 170 variants for recognition, synthesis, conversion, and separation.
Xolotl Tool
Public inference engine that streams experts across graphics memory, system memory, and storage; its documented 152 GiB DeepSeek weights were tested by the developer on a 24 GB RTX 4090 with 90 GB system memory. Performance is self-reported.
Whistle Tool
Earlier-release roundup: October 2 CPU speech model with a 16.9 MB download, up to 30-second clips, and seven European languages. Useful for compact offline integrations, with documented quality trade-offs.
Voodoo Dynamic Quant Tool
MIT-licensed tooling that learns per-tensor mixed-precision allocations and exports llama.cpp-compatible GGUF quantizations.
Von 1.0 Tool
An Apache-2.0 395M ModernBERT-based model and runtime for local structured decisions and HTTP serving.
Unsloth Muse Spark Open GGUF Tool
Published quantized files and llama.cpp commands for Muse Spark Open; listed variants remain hundreds of gigabytes and do not specify a complete runtime-memory budget.
Unsloth Tool
Toolkit and documentation for running and fine-tuning large open models faster and on smaller hardware, including aggressive dynamic quantization recipes that shrink models like GLM 5.2 by 80-plus percent while keeping most of their accuracy. The practical on-ramp to running near-frontier models privately.
Underdog Saluki 27B 1.0 Tool
Qwen-derived Apache-2.0 model offered as a 7.89 GB IQ2-mix GGUF for stock llama.cpp, with an optional separate vision add-on. Publisher-reported quality varies by task; running GPU-memory requirements are unstated in the dossier.
TurboQuant-MLX Tool
Quantization tooling for MLX with published size and speed measurements, including a 3-bit path that takes a 120-billion-parameter model from about 63GB down to 48GB on consumer Macs.
TensorSharp Tool
Public .NET inference engine for local model execution, including documented SSD-backed workflows; the specific 16 GB laptop demonstration remains unverified.
SwiftLM Tool
An MLX-based runtime for Apple silicon whose --stream-experts mode reads mixture-of-experts weights straight off an NVMe SSD, letting a machine run models several times larger than its RAM.
Survival LLM Tool
Open-source offline assistant and reference library with hardware-aware setup and an October 27 update; its first-aid evaluation is the maker’s own small test.
Strata Tool
MIT-licensed local Qwen inference engine with expert caching, a local interface and application-compatible APIs; large quantized configurations still need substantial RAM and storage.
SteerTTS Tool
An English speech-synthesis package with numeric controls for pitch, rate and other delivery features; MIT code and noncommercial weights, with documented quality limits.
SHADOW-50M Tool
A proof-of-concept 44M-parameter CPU model with exact arithmetic circuits, disk-backed retrieval and WebAssembly support.
RemoveMacAI Tool
A macOS 27 utility whose author says it removes AI model assets and installs a reversible profile to block their return; reclaim amounts vary and the code was not independently audited.
Pocket TTS Tool
Kyutai’s working speech-synthesis package adds tiny English checkpoints; code is MIT, weights are CC BY 4.0 with gated consent terms.
Pi 1.0 Tool
An extensible terminal coding agent with native MCP through Codemode, hosted and local model connectivity, and an MIT-licensed toolkit.
Ornith-1.0-9B GGUF Tool
The quantized build of the smallest member of the MIT-licensed Ornith-1.0 family, an open agentic coding line post-trained on top of Gemma 4 and Qwen 3.5. The Q4_K_M file is 5.63 gigabytes, which puts it within reach of a single consumer GPU.
OpenMed 3.0 Tool
A local-first developer toolkit for evidence-linked patient timelines and cohorts; the new brief walkthrough uses synthetic data and is not qualified for real-patient summaries.
Ollaya Tool
A public local runtime for typed, inspectable decision-model inference with a developer-oriented API.
Ollama 0.31 Tool
Run open models on your own computer; the new version nearly doubles Gemma's speed on Apple Silicon using multi-token prediction, on by default.
Nanbeige4.3-33B Tool
Apache-2.0 agent-model weights with official serving examples. A third-party Q4_K_M download is about 20.8 GB; the primary evidence does not state a total runtime GPU-memory requirement.
NVIDIA Nemotron 3.5 Transcribe Tool
Released offline transcription model supports 40 languages and recordings up to one hour; check the model card for execution instructions and OpenMDW-1.1 terms.
NVIDIA DGX Spark 64GB Tool
A 64GB configuration of NVIDIA's GB10 desktop AI system, sold by Acer, ASUS, Dell, Gigabyte, HP and MSI from October 23 starting at $4,999; NVIDIA says it runs models up to 100 billion parameters and two units can be clustered.
MicroLLM Lab Tool
A WebGPU browser lab for running and comparing several tiny quantized language models without an API key or local Python stack.
MiMo-V2.6-Distill-Qwen-9B Tool
Xiaomi's 9B Qwen3.5-based supervised distill for coding, agent, visual-coding, and cybersecurity research.
Mference Tool
Runs DeepSeek V4 Flash on Apple silicon by keeping the shared core, attention and cache resident while streaming each token's routed experts off the SSD. Publishes an unusually honest memory budget: about 3 GB working set, 90 to 98 GB on disk, tested at a 4,000-token context on a 24 GB Mac, with no quality parity test yet.
Liquid Playground Tool
Public hosted playground for trying Liquid AI models; a hosted test does not establish local-device compatibility.
Lean-QwenFlash Tool
Linux/NVIDIA runtime that streams Qwen expert weights from storage; its documented setup uses an approximately 85 GB download, a packed store, and a demonstrated 16 GB of GPU memory, with quality retention unmeasured.
Kyojin Tool
Public serving engine and scripts for a documented 128 GB AMD Strix Halo setup; the authors’ measurements are hardware-specific and the pack-conversion pipeline is absent.
KoboldCpp v1.124 Tool
The October 31 release adds YuE2 music-generation support and multiple built-in text-to-speech models through MusicUI; model hardware needs depend on the selected model.
Kev Tool
Apache-2.0 local Qwen-based models that return typed probability decisions for yes/no, choice, and rating tasks.
Kapitsa Tool
Apple Silicon Metal runtime for supported Qwen and NVIDIA Nemotron configurations; v0.1.36 is dated October 19. Performance comparisons are maintainer measurements.
Jeff Tool
Open code and downloadable small models for structured option selection, probabilities, and low-latency local decision loops.
Indra OCR Tool
Shipping CPU OCR package with an approximately 76 MB default model download; no GPU required. Code is Apache 2.0, while model weights have separate commercial restrictions.
H2O-Lightning-4B Tool
Apache-2.0 decision checkpoint with 9.1 GB of weights; the card reports local measurements on a 32 GB professional GPU, without establishing a minimum.
Gemma-4 12B Coder (GGUF) Tool
A fine-tuned, locally-runnable version of Google's Gemma-4 model specialized for programming tasks, packaged in a format that runs efficiently on everyday consumer hardware.
GLM-5.2 Tool
A flagship openly-available language model with a very large context window for long documents and code. Free to download and run yourself, with compressed versions for more modest hardware.
GLM 5.2 (GGUF, runnable locally) Tool
Zhipu AI's open, MIT-licensed mixture-of-experts model with a roughly million-token context, now packaged as ready-to-run quantized files you can host on your own machine. Strong on agent and coding workflows; this week it beat Claude on a narrow security benchmark at a fraction of the cost.
FLUX 3 Image Klein Tool
Downloadable generation and editing pipeline with base and distilled variants; hardware needs depend on configuration, and commercial use is subject to the FLUX Kommunity License.
DeepSeek-V4-Pro Tool
A downloadable 1.6-trillion-parameter mixture-of-experts model that activates 49 billion parameters per token, with a one-million-token context window under an MIT license. Serious server hardware required, but the weights are yours.
Comfy-Org Qwen Image 2.1 package Tool
ComfyUI-ready Qwen Image package with eight-step Turbo files in 16-bit and 8-bit formats; users must respect the underlying model license.
Cloudflare Clef and Clef-Flash Tool
Earlier-release roundup: October 1 Apache-2.0 models for scoring structured choices, including routing and classification, rather than generating prose. Published timing setup uses a single H200; consumer-hardware requirements are not established.
BotPsy GGUF Tool
Downloadable text-only tool-calling model for developers, with Q4 and other local-run variants.
Bonsai 2 Tool
Prism's downloadable ternary 27B model and runtime for local text, tool-calling and optional vision workflows.
BetterWispr Tool
A macOS dictation app offering on-device speech recognition and optional external endpoints; configured external providers receive the audio.
Basalt Tool
Public Blackwell-targeted inference engine for NVIDIA hardware, with expert offloading into system memory; community peak-throughput claims should not be read as average performance.
AlayaWorld Tool
Inference code and pretrained weights for an autoregressive world model with real-time camera control, prompt switching and long-horizon memory consistency, using an explicit 3D cache for spatial recall plus a compressed frame-history embedding. Training code is not included and the weights ship under a community license.