llama-cpp
llama.cpp merged Qwen's new architecture and a 97-gigabyte lookup table News
Support for Qwen3.8-Flash-Next landed in llama.cpp on August 27, adding a sparse-attention graph, vision, three quantizer fixes and machinery to stream a 97.7 GiB n-gram table that never has to sit on the GPU.
A llama.cpp fork is reviving $200 AMD cards nobody else supports News
A specialist fork of llama.cpp ships hand-written kernels for AMD's decade-old GFX906 architecture, making cheap used MI50 and Radeon VII cards usable for local inference, and upstream maintainers are now discussing porting the work back.
llama.cpp tagged v0.1.0 after ten thousand build numbers News
The most widely used local AI inference engine published its first semantic version tag on August 17, 2026, after years of releasing by build number, while a pending contribution adds a speculative-decoding mode that adjusts its own draft depth on the fly.
Two-bit models now run on every major llama.cpp backend News
The official 2-bit weight format merged into llama.cpp now works on mainline CPU, Metal, Vulkan, and CUDA after the CUDA path landed upstream, completing a backend sweep that started as an ARM-only contribution.
A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type News
An open pull request to llama.cpp tracks which mixture-of-experts submodels get used most during inference and promotes them to GPU memory on the fly, roughly doubling decode speed on an 8GB card in the author's own tests - while slowing other models down.
llama.cpp shipped DSpark for DeepSeek V4 Flash, and almost everyone called it the wrong name News
llama.cpp release b10228 merged speculative-decoding support for DeepSeek V4 Flash, but the new 0731 checkpoint embeds DSpark and ships no MTP at all, so the widely repeated "MTP support landed" advice points users at a head their model does not contain.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
llama.cpp ships the fix that lets DeepSeek V4 Flash call tools mid-thought News
DeepSeek's new open-weight model shipped without a standard chat template, and local agents kept dying when it emitted a tool call inside its reasoning; llama.cpp release b10217 and KoboldCpp v1.118 both landed fixes on 1 August.
llama.cpp Merges MiniMax M3's Sparse Attention, Because Running It Dense Gives Wrong Answers News
Support for MiniMax M3's block-sparse attention landed in llama.cpp today, and the contributor is explicit that it is not a speed optimisation: the model was trained sparse, so running it dense produces degraded output.
llama.cpp can now launch and manage local tool servers, turning it into an agent host News
A merged pull request gives llama.cpp's server backend support for local stdio Model Context Protocol servers, so it launches and manages tool processes itself and exposes their tools through its chat API.
llama.cpp-gfx906 Tool
A llama.cpp fork with hand-written kernels for AMD's GFX906 architecture, making used Instinct MI50, MI60, and Radeon VII cards usable for local inference. Ships custom flash-attention, RoPE, and matrix-multiply paths plus overclocking and power-scaling scripts.
llama.cpp v0.1.0 Tool
The engine behind most local AI setups published its first semantic-looking version tag on August 17, 2026, pinned to commit 7c35571. Useful mainly to packagers and anyone who needs a version string a dependency resolver understands; the project still makes no API-stability promise.
jlens-gguf Tool
A GGUF-native implementation of Anthropic's Jacobian Lens for local models, with a browser UI to visualize, swap, and ablate a model's internal concepts live as it generates through llama.cpp.
Voodoo Quant Tool
A per-tensor sensitivity-aware quantization method that spends more bits on important tensors, claiming large divergence reductions over standard llama.cpp and Unsloth quants, especially at 1-bit and 2-bit; GGUF files run in unmodified llama.cpp.
Qwen3.8-Flash-Next GGUF (Unsloth) Tool
Unsloth's quantized GGUF conversions of Qwen3.8-Flash-Next, including a 2-bit UD-Q2_K_XL build at roughly 78.9 GB across three shards -- about a fifth of the official bf16 repository. The model card carries working setup instructions for llama.cpp, vLLM and Ollama.
Muse Glimmer 30B GGUF (Unsloth) Tool
Community-packaged quantized builds of Muse Glimmer that fit under 20 GB, with setup instructions for llama.cpp, Ollama, vLLM, and SGLang. This is the practical path if you want the model running on a single 24 GB consumer graphics card rather than compiling the full-precision weights yourself.
CachyLLama Tool
MIT-licensed llama.cpp fork that saves conversation and system-prompt caches to SSD and restores them after a restart, so local agents stop reprocessing the same prompt prefix every turn. Its own benchmark reports long repeated agent prefixes going from minutes cold to about a second warm.