gguf
llama.cpp merged Qwen's new architecture and a 97-gigabyte lookup table News
Support for Qwen3.8-Flash-Next landed in llama.cpp on August 27, adding a sparse-attention graph, vision, three quantizer fixes and machinery to stream a 97.7 GiB n-gram table that never has to sit on the GPU.
A 22-gigabyte local coder matched Opus on a 25-problem slice News
TielCoder, a 4-bit re-quantization of the Ornith-1.5 mixture-of-experts model, fits in 22.4 GB and fixed 12 of 25 recent real-world coding issues in its maintainer's tests, the same count as Claude Opus 4.6 at medium effort.
Qwen3.8-27B's compression data has a blind spot where its speed-up head lives News
The importance matrix used to compress Qwen3.8-27B contains no entries for block 64, the model's multi-token-prediction head, because that head never activates during the standard calibration run, which is where the most aggressive one-bit builds start to break.
How a model is stored: safetensors, GGUF, and why one model arrives in 96 files Lesson
A trained model is just a large dictionary of numbered arrays saved to disk, and the file format that holds them determines whether the model loads safely, loads fast, and loads at all on your hardware.
Unsloth DeepSeek-V4-Flash-0731 GGUF Tool
Published quantisations of DeepSeek's 671-billion-parameter Flash model, ranging from roughly 91 GB at two bits to 162 GB at eight. The card is also the clearest available statement of what hardware each tier actually needs.
TielCoder 35B-A3B GGUF Tool
4-bit dynamic re-quantization of Ornith-1.5-35B-A3B for llama.cpp. The benchmarked 22.4 GB tier fits a 24 GB card and fixed 12 of 25 live software issues in the maintainer's tests. Includes a vision projector for reading screenshots and stack traces.
Ternary-Bonsai-27B (GGUF) Tool
PrismML's ternary-weight 27B model in GGUF at ~7.2 GB deployed, with custom CUDA/Metal/CPU kernels. Expands local hardware reach, though agentic reliability is still limited per early tests.
Qwen3.8-Flash-Next GGUF quants Tool
Unsloth's quantized builds of Qwen's newest architecture, in eleven sizes from roughly 72.5 GB at the smallest to about 354 GB at full precision. No official VRAM figure is published; community reports run a 4-bit build on a 16 GB card with around 100 GB of combined system memory.
Qwen3.8-Flash-Next GGUF (Unsloth) Tool
Unsloth's quantized GGUF conversions of Qwen3.8-Flash-Next, including a 2-bit UD-Q2_K_XL build at roughly 78.9 GB across three shards -- about a fifth of the official bf16 repository. The model card carries working setup instructions for llama.cpp, vLLM and Ollama.
Qwen3.8-27B GGUF builds Tool
Ready-to-run compressed builds of Alibaba's 27-billion-parameter multimodal Qwen3.8, covering the full ladder from eight-bit down to one-bit. Community testing points to the six-bit build, around 22 gigabytes, as the conservative floor for serious agentic coding, with three-bit still usable and one-bit rebuilds degrading sharply because the calibration file has no data for the model's multi-token-prediction head.
Ornith-1.0-9B GGUF Tool
The quantized build of the smallest member of the MIT-licensed Ornith-1.0 family, an open agentic coding line post-trained on top of Gemma 4 and Qwen 3.5. The Q4_K_M file is 5.63 gigabytes, which puts it within reach of a single consumer GPU.
Muse Glimmer 30B GGUF (Unsloth) Tool
Community-packaged quantized builds of Muse Glimmer that fit under 20 GB, with setup instructions for llama.cpp, Ollama, vLLM, and SGLang. This is the practical path if you want the model running on a single 24 GB consumer graphics card rather than compiling the full-precision weights yourself.
KoboldCpp v1.118 Tool
Single-binary local model server that shipped its own fix for multi-turn DeepSeek V4 Flash prompt-processing problems on the same day as the llama.cpp fix. Useful if you want a working DS4 setup without building anything.
Inkling-Small GGUF Tool
Quantized builds of Thinking Machines' newly released 276B/12B multimodal open-weight model, packaged for llama.cpp, LM Studio and Ollama so you do not have to download the 532GB original.
Gemma-4 12B Coder (GGUF) Tool
A fine-tuned, locally-runnable version of Google's Gemma-4 model specialized for programming tasks, packaged in a format that runs efficiently on everyday consumer hardware.
DeepSeek-V4-Flash-0731 GGUF (Unsloth) Tool
Community quantizations of the new MIT-licensed DeepSeek weights in GGUF form, running from roughly 83GB at aggressive low precision to about 162GB at 8-bit. Usable on high-memory workstations and multi-GPU rigs, not on a laptop.