News · 2026-08-26

Qwen put a 20-million-entry n-gram table inside a model

Alibaba's Qwen team released Qwen3.8-Flash-Next, which it describes as an experimental preview of the architecture that will underpin Qwen4. The model has 125 billion parameters with only 6 billion active per token -- plus 51 billion parameters sitting in a lookup table of 20 million word pairs and triples. That table is the bet: Qwen is arguing that as AI accelerators stay starved for memory rather than compute, the cheapest place to add parameters is the place you can move off the GPU.

Key facts

• 125B parameters with 6B activated, plus 51B of n-gram embeddings and 4B in a multi-token-prediction head; 512 experts with 10 routed plus 1 shared per token.
• 20,000,000 n-gram entries -- bigrams and trigrams, indexed at layer 2. Context is 262,144 tokens natively, extensible to 1,000,000.
• Released August 26, 2026 under the Qwen Community License 1.0; the official repository ships 360 GB of bf16 weights.
• Primary sources: the Qwen/Qwen3.8-Flash-Next model card, the technical report and repo, and Qwen's blog post.

The hook. N-grams are the oldest trick in language modelling -- count how often words follow other words, and predict from the counts. That approach was declared obsolete when neural networks arrived. Qwen just welded a 20-million-entry version of it into a frontier model and made it 51 billion parameters wide.

Background. Qwen's last flagship, Qwen3.8-Max, shipped as a paid API with no weights, and the smaller Qwen3.8-27B came with a restrictive contract. So the first question the open-weights community asked about this release was about the licence, not the architecture.

What they did. Four changes, all aimed at the same target. The attention stack pairs Gated DeltaNet blocks with a new Qwen Sparse Attention that selects context at the micro-block level rather than token by token, which is what cuts long-context latency. A Gated Residual mechanism adds four branches at bottleneck rank 320 with data-dependent read gates and per-branch write gates on widened residual streams. The training recipe applies the Muon and AdamW optimizers to different weight categories and, guided by refitted scaling laws, eliminates batch-size warmup entirely by starting at the target batch size.

And then the n-gram table. In Qwen's own words: "Embeddings provide a unique axis for parameter scaling that requires less computation and is more amenable to offloading than Mixture-of-Experts. By indexing with short n-grams, this approach makes parameter scaling highly efficient for memory-constrained accelerators without sacrificing quality."

How it works. A mixture-of-experts layer adds parameters by adding specialists, but a specialist has to actually run -- its weights must be on the GPU when its turn comes, and which one gets picked is unpredictable. An embedding table adds parameters by adding rows, and a row is retrieved, not computed. It is the difference between hiring more staff and buying a bigger filing cabinet. The cabinet can go in the next room; the staff cannot. Because a lookup is just an address, that 51-billion-parameter table can sit in system memory or on an SSD and be paged in as needed -- exactly the trick described in offloading and streaming weights.

The licence, answered. The Qwen Community License 1.0 grants the right to use, copy, modify, distribute, sublicense, sell, deploy, host, fine-tune and create derivative works. Two conditions: products with more than 100 million monthly active users or $20 million monthly revenue must display the model name prominently in the interface; and any licensee running a "Model as a Service" business -- giving third parties API or hosted-endpoint access -- or an "AI Work Assistant" business, defined as a product primarily for AI-assisted coding or office productivity, must obtain a separate licence first. Internal use is explicitly exempt. So the weights are genuinely open, with a carve-out aimed squarely at competitors, and it is a materially better deal than Qwen3.8-Max got.

Why it matters. The release hit 611 points and 197 comments on Hacker News within twelve hours, the hardest engagement number of the day, with a mod-pinned r/LocalLLaMA megathread reporting it outperforming DeepSeek V4 Flash at a fraction of the parameter count. Qwen frames the whole thing as a thesis rather than a product: "Sustainable progress toward artificial general intelligence (AGI) that benefits everyone demands architectural innovation."

The honest caveat. The name says Flash and the headline says 6 billion active parameters, and both make this sound small. It is not. Every one of the 512 experts and the entire n-gram table must be resident or paged; the official download is 360 GB in bf16. Low active-parameter counts buy throughput, not a smaller machine -- see why LLM inference is memory-bound. Practical local paths do exist: Unsloth's GGUF build ships a 2-bit variant at roughly 78.9 GB across three shards, with llama.cpp and vLLM instructions. And every comparison number in the model card is Qwen's own, including the column labelled Claude Opus 4.6 -- independent retests have not landed yet.

Primary source, verified: read the paper →

Ground Truth. (2026, August 26). Qwen put a 20-million-entry n-gram table inside a model. Ground Truth. https://groundtruth.day/news/qwen-put-a-20-million-entry-n-gram-table-inside-a-model.html

@misc{groundtruth:qwen-put-a-20-million-entry-n-gram-table-inside-a-model,
  title  = {Qwen put a 20-million-entry n-gram table inside a model},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/qwen-put-a-20-million-entry-n-gram-table-inside-a-model.html}
}

Topics: open-weights · architecture · china · mixture-of-experts · efficiency · models · licensing