News · 2026-08-07

Two-bit models now run on every major llama.cpp backend

The official 2-bit weight format in llama.cpp now runs on mainline CPU, Metal, Vulkan, and CUDA, after the CUDA path merged upstream and was documented on August 4. That completes a backend sweep for a format that entered the project a month earlier as an explicitly ARM-only contribution, and it is the step that moves 2-bit quantization from a fork curiosity to something you can actually deploy across ordinary hardware.

Key facts

• The originating pull request, ggml: add Q2_0 quantization support (CPU), merged on July 7, 2026.
• The official format packs each weight into two bits with one half-precision scale per group of 64 weights, at 2.25 bits per weight.
• The maintainers' status page now states that the format "now runs on mainline CPU, Metal, Vulkan, and CUDA," with the CUDA merge documented on August 4.
• Primary sources: the merged pull request and the Bonsai status page.

Start with what quantization buys. A model's weights are normally stored at 16 bits each. Cutting that to two bits shrinks the file roughly eightfold, which is the difference between a model that fits in your machine's memory and one that does not. The catch has always been quality: throw away that much precision and the model usually gets noticeably worse. The interesting claim attached to this format is that, for models trained with it in mind, it largely does not - the pull request's own measurements report very small divergence from half precision, with the model's top choice agreeing with the unquantized version in the high 99 percent range on an 8B model.

The part of this story that is more instructive than the compression ratio is the format negotiation, which is visible in the pull request in plain sight. The contributing team, PrismML-Eng, builds ternary models whose weights natively use groups of 128. Upstream asked for groups of 64. Contributor khosravipasha wrote in the pull request: "Our models natively support group size 128; however, it was requested to do group size 64 for the official Q2_0 format, so this PR uses 64." The team kept its own packing alive as a separate sibling format in its fork, noting that the extra 0.125 bits per weight becomes significant on larger models.

That is not bikeshedding. It is the moment a format stops being one lab's file layout and becomes a standard, and it has a real cost for anyone who moved early: files in the fork's group-128 packing will not load on mainline, because the two formats share a type identifier while using different block sizes. The maintainers' own status page is blunt about the migration, warning against depending on the reserved future format name and stating that the group-64 files are the ones mainline reads. If you have 2-bit GGUFs from before this landed, check which packing they use before assuming they still work - it is the same class of trap as the safetensors and GGUF format differences that catch people moving models between runtimes.

The backend sweep is what makes the standardization worth caring about. The original merge was, in the contributor's words, "CPU only (ARM NEON + generic scalar fallback)," with a note that "we have the x86, Metal, CUDA, and Vulkan backends ready to submit later." A month on, most of that promise has been delivered upstream. A quantization that only runs on Apple silicon is a demo. One that runs on Nvidia GPUs, on Vulkan, on Metal, and on ARM CPUs is a deployment target, and it is the difference between an interesting compression result and something that changes what hardware a given model needs.

The honest caveats are two. First, all the quality numbers here come from the team that makes the models, measured on their own models - an independent evaluation on third-party weights has not been published. Second, 2-bit quality holds up best for models trained or prepared with the format in mind; naively crushing an arbitrary 16-bit model to two bits is a different and much less flattering experiment. Separately, a pull request routing quantized KV-cache decode to a faster kernel path on Intel's Battlemage GPUs remains open, which is a reminder that cache quantization is a distinct axis from weight quantization - and one that, as we have covered, changes which tokens the model picks.

Primary source, verified: read the paper →

Ground Truth. (2026, August 7). Two-bit models now run on every major llama.cpp backend. Ground Truth. https://groundtruth.day/news/two-bit-models-now-run-on-every-major-llama-cpp-backend.html

@misc{groundtruth:two-bit-models-now-run-on-every-major-llama-cpp-backend,
  title  = {Two-bit models now run on every major llama.cpp backend},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/two-bit-models-now-run-on-every-major-llama-cpp-backend.html}
}

Topics: open-source · local-inference · quantization · llama-cpp · efficiency · tools