Learn · Beginner

N-gram language models

An n-gram language model predicts the next word by counting. Take a large pile of text, count how often each word follows each preceding word or pair of words, and use those counts as probabilities. That is the whole idea. It is the simplest thing that can honestly be called a language model, it powered speech recognition and machine translation for roughly thirty years, and every design decision in modern large language models is a response to a specific way it failed.

The core assumption

A language model assigns a probability to a sequence of words. Doing that exactly requires knowing the probability of each word given everything before it, which is impossible -- most long sentences have never been written before, so there is nothing to count.

N-gram models make a deliberate simplification: assume the next word depends only on the previous few. A bigram model looks back one word, a trigram looks back two, a 5-gram looks back four. Formally this is a Markov assumption. Practically it means the model estimates the chance of "mat" following "sat on the" by dividing how many times "sat on the mat" appeared by how many times "sat on the" appeared. That is called a maximum likelihood estimate, and it is arithmetic, not learning in any modern sense.

The classic reference treatment is Chapter 3 of Jurafsky and Martin's Speech and Language Processing, which remains the clearest walkthrough anyone has written.

The problem that consumed a decade of research

Count-based estimates break on anything you have never seen. If "purple bureaucratic hamster" does not appear in your corpus, the model assigns it probability zero -- and because sentence probabilities multiply, one zero makes an entire perfectly reasonable sentence impossible. As you grow n from 2 to 5, the model gets sharper and the zeros get vastly more common, because there are astronomically more possible 5-word sequences than 2-word ones.

The fix is smoothing: move a little probability mass away from what you saw and give it to what you did not. The crudest version adds one to every count. The good versions are cleverer. Backoff falls back to a shorter n-gram when the longer one is unseen. Interpolation always blends all the orders together. The best-performing classical method, Kneser-Ney smoothing, adds a twist worth understanding: when estimating how likely a word is in a novel context, it does not use how often the word appears, but in how many distinct contexts it appears. "Francisco" is common, but almost always after "San," so it is a bad bet in a new context. Chen and Goodman's 1996 empirical study compared these systematically and settled the field.

The way these models were judged is a metric still used today: perplexity, roughly the number of equally likely options the model thinks it is choosing among at each step.

Why they were replaced

Two failures, and both are the reason modern architectures look the way they do.

No generalization across words. To an n-gram model, "dog" and "puppy" are unrelated symbols with unrelated counts. Millions of examples of one teach it nothing about the other. The fix was to represent each word as a vector of numbers positioned so similar words sit close together, which lets evidence about one word inform predictions about its neighbours. Bengio and colleagues' 2003 neural probabilistic language model introduced exactly this -- the ancestor of every embedding in use today.

No long-range memory. A 5-gram model cannot know that a sentence started with "The keys that were on the table" and therefore needs "were," not "was." Fixing that required architectures that carry information forward -- recurrent networks first, then transformers, whose whole contribution is letting any position attend directly to any other.

Scale did not save the counting approach. Google's 2007 machine translation work trained n-gram models on two trillion tokens and showed quality improving steadily with more data -- an early scaling result -- and it still lost to neural models, because more counts do not buy generalization.

Where n-grams are still alive

They never actually left.

• Evaluation metrics. BLEU and ROUGE, the standard scores for translation and summarization, compare overlapping n-grams between output and reference.
• Lexical search. Keyword retrieval systems including BM25 work on term and phrase statistics.
• Data hygiene. Training-data deduplication and benchmark contamination checks are largely n-gram matching at scale.
• Fast heuristics. Spelling correction, keyboard autocomplete, and speculative decoding drafters use cheap n-gram statistics where a neural forward pass would be too slow.
• Memorization research. Infini-gram indexed a trillion tokens of unbounded n-grams to study what neural models reproduce verbatim.

And in 2026 the idea came back inside the architecture. Alibaba's Qwen shipped a model with a 20-million-entry n-gram embedding table welded into the network, holding 51 billion parameters. The reasoning is a direct descendant of everything above: a lookup table is memory rather than computation, so unlike a mixture-of-experts layer it can sit off the accelerator and be paged in. The oldest trick in the field turns out to be the cheapest way to add parameters to the newest models.

What to take away

The lesson is not that counting is obsolete. It is that counting is a lower bound you should always know. If a new architecture cannot beat a well-smoothed 5-gram on your data, something is wrong with your setup, not with n-grams. And understanding why they fail -- no sharing between similar words, no memory past a fixed window -- is the fastest route to understanding why embeddings, tokenization choices, and attention exist at all. For how the counting intuition maps onto what a modern model does at each step, see how AI picks its next word.

Ground Truth. (2026, August 26). N-gram language models. Ground Truth. https://groundtruth.day/learn/n-gram-language-models.html

@misc{groundtruth:n-gram-language-models,
  title  = {N-gram language models},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/n-gram-language-models.html}
}

Topics: fundamentals · language-models · history · statistics · smoothing