News · 2026-07-26

llama.cpp Merges MiniMax M3's Sparse Attention, Because Running It Dense Gives Wrong Answers

Support for MiniMax M3 landed in llama.cpp today, and the notable part is why it needed special handling at all. The model uses block-sparse attention that it was trained with, not bolted onto afterwards - so the usual fallback of running the model with ordinary dense attention does not merely run slower, it produces measurably worse output. Contributor timkhronos states it directly in the pull request: "MSA is not an optional speed optimization, as the model is trained with sparse attention, so block selection is part of the model's semantics."

Key facts

• Pull request 24908 merged on July 26, 2026, adding 1,044 lines across 21 files.
• The model is a 60-layer mixture-of-experts design with 128 experts, of which three layers are dense and 57 are sparse.
• Sparse attention holds decode cost at roughly 2,048 key-value tokens regardless of context length.
• Primary source: the merged pull request; the mechanism is documented in arXiv:2606.13392.

Here is what the mechanism actually does. In ordinary attention, generating each new word means consulting every token that came before it, which is why long conversations get progressively slower and hungrier for memory - the KV cache grows without bound. MiniMax's approach adds a lightweight indexer that scores the stored context, pools those scores into blocks of 128 tokens, and picks the best 16 blocks, always including the block the model is currently writing in. The main attention then runs over just those. Sixteen blocks of 128 tokens is 2,048 tokens of context actually attended to, whether the conversation is five thousand tokens long or a million.

The librarian analogy is apt. Dense attention re-reads every book in the library before answering each question. This design keeps a card catalogue, checks it, pulls sixteen shelves, and reads those. The catalogue lookup is cheap; the reading is bounded. What makes it different from ordinary sparse attention tricks is that MiniMax trained the model this way from the start, so the model's learned behaviour assumes it will be reading sixteen shelves. Hand it the whole library and it does not get smarter - it gets confused, because that is not the input distribution it learned in.

That distinction drove the engineering. Rather than the usual approach of implementing a fast path and quietly falling back to dense attention whenever the fast path is unavailable, the pull request implements the block scoring and selection in both prefill and decode, mapping the four attention groups onto a broadcast dimension so a single grouped flash-attention call handles them. The dense path survives only as an escape hatch for configurations that cannot run sparse. The contributor is careful to note it is "an out-of-distribution approximation that degrades output quality."

The subtler point that came up in review concerns quantization. The indexer projections are tiny, but they decide which blocks get read. Quantize them too aggressively and you do not get slightly noisier attention scores; you get a different set of blocks selected, meaning the model reads entirely different context. It is a discrete error, not a gradual one - the difference between a slightly blurry map and a map to the wrong building. The recommendation is to keep those small projections at high precision during conversion, and it is exactly the kind of detail that gets lost when a model passes through several hands on the way to a downloadable file.

The current limits are worth knowing before anyone runs this in production. Sparse operation requires flash attention. A unified key-value cache with parallel sequences falls back to dense. Context shifting is unsupported. Quantized cache modes had not been validated at merge time. And vision support - MiniMax M3 accepts images and video - is a separate pull request that remains open, with one approval and a second required review outstanding. That one implements the model's less standard vision details, including three-axis positional encoding across time, height and width, and a two-stage patch merger.

MiniMax positions M3 as combining coding and agentic ability, a million-token context, and native image and video input, in its launch post. Its model card publishes the weights under a license labelled minimax-community, which makes it open-weight rather than open-source in the strict sense - a distinction that keeps mattering as more labs ship portable weights under custom terms.

The honest caveat is that the performance claims attached to this merge are a developer's own measurements on their own hardware, not an independent benchmark, and the reviewers flagged the key-value cache design as an area they may want to revisit. But the compatibility work itself is done and public, which is more than can be said for most models announcing million-token contexts this month. When Kimi K3's weights arrive, this is the kind of unglamorous plumbing that will determine whether anyone outside a data centre can actually use them.

Primary source, verified: read the paper → (arXiv 2606.13392)

Ground Truth. (2026, July 26). llama.cpp Merges MiniMax M3's Sparse Attention, Because Running It Dense Gives Wrong Answers. Ground Truth. https://groundtruth.day/news/llama-cpp-merges-minimax-m3s-sparse-attention.html

@misc{groundtruth:llama-cpp-merges-minimax-m3s-sparse-attention,
  title  = {llama.cpp Merges MiniMax M3's Sparse Attention, Because Running It Dense Gives Wrong Answers},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/llama-cpp-merges-minimax-m3s-sparse-attention.html}
}

Topics: open-weights · llama-cpp · sparse-attention · local-inference · minimax