quantization
Qwen3.8-27B shares its predecessor's bones, but not its contract News
Alibaba's Qwen3.8-27B shipped with the same coarse architecture as Qwen3.6-27B, prompting accusations it was a relabel with knowledge stripped out, but the published comparison shows knowledge scores flat or slightly up.
Meta ships a 30B agent model that fits on one gaming GPU News
Meta released Muse Glimmer, a 30-billion-parameter open-weight agent model under the permissive Apache 2.0 license that shrinks to under 20 GB once quantized, small enough to run on a single 24 GB consumer graphics card.
Two-bit models now run on every major llama.cpp backend News
The official 2-bit weight format merged into llama.cpp now works on mainline CPU, Metal, Vulkan, and CUDA after the CUDA path landed upstream, completing a backend sweep that started as an ARM-only contribution.
The 'Ternary' 20B Model Everyone Downloaded Today Ships as a Two-Bit Package News
DeepGrove's Maple-Preview is being shared as a 20-billion-parameter ternary model that fits in about 5 gigabytes, but its native release is a 40GB BF16 repository and the small footprint belongs to a separate two-bit MLX build with no published ternary training recipe.
Liquid Shipped a 2.6B Tool-Calling Model and Told You Not to Code With It News
Liquid AI released LFM2.5-2.6B, a small model whose stack is mostly cheap convolution layers with attention rationed to eight of thirty, built for tool use and long context - and whose own model card says not to use it for agentic coding or knowledge-heavy work.
The Cheap 284B Rig Is Really 768GB of Server Memory News
A builder running DeepSeek V4-Flash at 33 tokens a second on two RTX 3090s is holding about 6.6GB of weights per card and roughly 170GB per instance in system memory on a four-socket enterprise server, which is where the model actually lives.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
llama.cpp ships the fix that lets DeepSeek V4 Flash call tools mid-thought News
DeepSeek's new open-weight model shipped without a standard chat template, and local agents kept dying when it emitted a tool call inside its reasoning; llama.cpp release b10217 and KoboldCpp v1.118 both landed fixes on 1 August.
A 26-billion-parameter model runs in 2GB of RAM by streaming experts off the SSD News
TurboFieldfare, an open-source Swift and Metal runtime, runs Gemma 4's 26-billion-parameter model on an 8GB MacBook Air by keeping only a 1.35GB core in memory and pulling each token's experts from disk as it needs them.
DeepSeek V4 Flash hits 32 tokens a second on one desktop News
A published benchmark shows DeepSeek's 284-billion-parameter V4 Flash generating 32 tokens per second entirely on one AMD Strix Halo machine, using aggressive quantization, speculative decoding and reduced expert routing.
Kimi K3 is downloadable, but the floor to run it is eight datacenter GPUs News
Kimi K3's 1.56-terabyte checkpoint needs a single eight-GPU B300 or MI355X node as its practical minimum, and no version of llama.cpp can load it today, so open weights currently mean operator-scale rather than local.
How a model is stored: safetensors, GGUF, and why one model arrives in 96 files Lesson
A trained model is just a large dictionary of numbered arrays saved to disk, and the file format that holds them determines whether the model loads safely, loads fast, and loads at all on your hardware.
Poolside's Laguna S 2.1 shipped with a broken chat template - and the fixes explain the reviews News
Days after releasing its open coding model, Poolside has been repairing it in public: the base chat template shipped with reasoning disabled by default and a 32,768-token generation cap, and its own quantised builds needed re-releases to fix tool calls and thinking.
KronQ makes 2-bit models work where the standard method collapses News
Adding gradient information to model compression takes a 2-bit LLaMA-3-70B from complete gibberish under the dominant method to a functioning model, suggesting the standard approach has been missing half the picture since 2022.
Bonsai puts a 27B model on your phone, and shows what breaks News
PrismML shrank a 27-billion-parameter model from 54 gigabytes to 3.9 by storing each weight as a single bit, and published the benchmark table showing tool calling and vision falling off a cliff.
Nvidia's 'Puzzle' pipeline shrinks a 120B model to 75B and roughly doubles serving throughput News
Nvidia's Nemotron-3-Puzzle compresses a 120-billion-parameter model down to 75 billion using a multi-stage pipeline of pruning, distillation, and quantization -- roughly doubling throughput on a B200 node and lifting single-H100 concurrency at long context from one request to eight.
A model that rivals the frontier now squeezes onto a single high-end desktop News
Aggressive compression shrinks GLM 5.2 by more than 80 percent while keeping most of its accuracy, putting a near-frontier model within reach of local hardware.
Quantization: Shrinking AI Models to Run on Modest Hardware Lesson
Storing a model's numbers with less precision - 8, 4, or even fewer bits instead of 16 - makes it dramatically smaller and faster, often with almost no loss in quality. It's why big models can run on a laptop or a single GPU.
bitnet.cpp Tool
Microsoft's official inference framework for 1.58-bit ternary language models, built on llama.cpp with optimized CPU and GPU kernels for running very heavily compressed models on ordinary hardware.
Xiaomi MiMo-V2.5-DFlash Tool
Xiaomi's official DFlash release on Hugging Face -- a 1-trillion-parameter mixture-of-experts model (42B active) under an MIT license, with FP4 quantization and parallel decoding for high inference throughput.
Voodoo Quant Tool
A per-tensor sensitivity-aware quantization method that spends more bits on important tensors, claiming large divergence reductions over standard llama.cpp and Unsloth quants, especially at 1-bit and 2-bit; GGUF files run in unmodified llama.cpp.
Unsloth Qwen3.8-27B GGUF Tool
Quantized builds of Alibaba's newest 27B open-weight model, published within minutes of the release, in a range of sizes that fit on a single consumer graphics card.
Unsloth Kimi-K3-GGUF Tool
Converted local-inference builds of Moonshot's Kimi K3: a 1.51 TB four-bit UD-Q4_K_XL file, a 1.56 TB eight-bit build, and BF16/F16/F32 multimodal projector files that preserve an image-input path. Datacenter-scale hardware still required.
Unsloth DeepSeek-V4-Flash-0731 GGUF Tool
Published quantisations of DeepSeek's 671-billion-parameter Flash model, ranging from roughly 91 GB at two bits to 162 GB at eight. The card is also the clearest available statement of what hardware each tier actually needs.
Unsloth Tool
Toolkit and documentation for running and fine-tuning large open models faster and on smaller hardware, including aggressive dynamic quantization recipes that shrink models like GLM 5.2 by 80-plus percent while keeping most of their accuracy. The practical on-ramp to running near-frontier models privately.
TurboQuant-MLX Tool
Quantization tooling for MLX with published size and speed measurements, including a 3-bit path that takes a 120-billion-parameter model from about 63GB down to 48GB on consumer Macs.
Ternary-Bonsai-27B (GGUF) Tool
PrismML's ternary-weight 27B model in GGUF at ~7.2 GB deployed, with custom CUDA/Metal/CPU kernels. Expands local hardware reach, though agentic reliability is still limited per early tests.
Ternary Bonsai models Tool
A family of 1.7B, 4B and 8B models built for extreme quantization, shipped in the official group-64 two-bit format that mainline llama.cpp reads. Useful if you want to see what 2-bit inference feels like without converting anything yourself.
Poolside Laguna S 2.1 (GGUF) Tool
Open-weight 118B mixture-of-experts coding agent activating about 8B parameters per token, under the permissive OpenMDW-1.1 licence, in GGUF plus FP8, NVFP4 and INT4 builds. Use the current re-released Q4/Q8 files - the initial ones shipped with a broken chat template.
Muse Glimmer 30B GGUF (Unsloth) Tool
Community-packaged quantized builds of Muse Glimmer that fit under 20 GB, with setup instructions for llama.cpp, Ollama, vLLM, and SGLang. This is the practical path if you want the model running on a single 24 GB consumer graphics card rather than compiling the full-precision weights yourself.
Maple-Preview (2-bit MLX build) Tool
DeepGrove's 20B mixture-of-experts model with about 1B active parameters per token, packaged for Apple Silicon at roughly 5.3GB. The build uses affine two-bit group quantisation with four-bit embeddings and output head, and its loader packs ternary values into two-bit codes. Note that the native BF16 repository is about 40.4GB, and DeepGrove publishes no ternary training recipe or independent evaluation.
LvLLM Tool
A community inference runtime specialised in hybrid CPU and GPU execution of mixture-of-experts models, with NUMA-aware scheduling, expert weight management, and MXFP4 quantization kernels. Its DeepSeek V4 build publishes a working dual-RTX-3090 launch configuration at 22K context.
Inkling-Small GGUF Tool
Quantized builds of Thinking Machines' newly released 276B/12B multimodal open-weight model, packaged for llama.cpp, LM Studio and Ollama so you do not have to download the 532GB original.
DeepSeek-V4-Flash-0731 GGUF (Unsloth) Tool
Community quantizations of the new MIT-licensed DeepSeek weights in GGUF form, running from roughly 83GB at aggressive low precision to about 162GB at 8-bit. Usable on high-memory workstations and multi-GPU rigs, not on a laptop.
Bonsai 27B (GGUF) Tool
PrismML's roughly 27.8-billion-parameter Qwen-derived model trained with 1-bit binary or 1.58-bit ternary weights end to end, which the company says fits in about 4 GB and runs on phone-class hardware. Performance figures are vendor-reported and not independently replicated.
Bonsai 27B Tool
PrismML's 1-bit and 1.58-bit builds of Qwen3.6 27B, compressing a 54 GB model to 3.9 GB (binary) or 5.9 GB (ternary) and running at roughly 11 tokens per second on an iPhone 17 Pro. The release ships an honest benchmark table showing the cost: instruction following, tool calling, and vision all degrade sharply, and the vendor states agentic coding is not a strong target of this release.