Learn · Intermediate

Multi-Token Prediction: Teaching Models to Look Several Words Ahead

Multi-token prediction (MTP) trains a language model to predict several upcoming tokens at once instead of just the next one. It matters because that extra look-ahead gives the model a richer training signal -- forcing it to plan a few words rather than greedily guessing one -- and it doubles as a built-in way to make the finished model generate text faster. Meta's 2024 paper by Gloeckle and colleagues and DeepSeek's V3 report both showed it improving quality and speed together, and by 2026 it is a standard ingredient in efficiency-focused open models like Alibaba's Qwen3-Next.

The problem with predicting one token at a time

A standard language model is trained on a single objective: given everything so far, predict the next token. This is simple and it works, but it is myopic. The model is only ever graded on the immediate next word, so it can learn shortcuts -- local patterns that make the next token easy but do not require it to understand where the sentence is going. If a passage needs the model to commit to an idea that only pays off three words later, next-token training barely rewards that foresight.

Multi-token prediction changes the grading. During training, from each position the model must predict not just the next token but the next several -- say the next four -- using a few small extra output "heads" that branch off the shared body of the transformer. The main network still does the heavy lifting of understanding the context; each head is a thin layer that reads that shared understanding and guesses token 1, token 2, token 3, and token 4 ahead. Because getting token 4 right requires the model to have actually planned that far, the objective pushes the representation to encode more about the future, not just the immediate next step.

An analogy

Next-token prediction is like sight-reading music one note at a time -- you can stumble through, but you are always reacting. Multi-token prediction is like being asked to hum the next whole bar: you cannot do it without holding the phrase's direction in your head. Training on the harder task builds the internal sense of where things are going, and that sense makes even the one-note-at-a-time performance better.

Two payoffs: better training, faster inference

The first payoff is quality. The denser signal -- several predictions per position instead of one -- teaches the model more per token of data, which is especially valuable as high-quality training text gets scarce. Gloeckle and colleagues reported that models trained this way improved on generative and coding tasks, with the benefit growing at larger model sizes.

The second payoff is speed, and it reuses the same extra heads. Normally a model generates one token, appends it, and runs the whole network again for the next -- one expensive forward pass per token. With trained multi-token heads, the model can propose several tokens in a single pass and then verify them cheaply, a technique closely related to speculative decoding. When the guesses are accepted, you get multiple tokens for roughly the cost of one, which is why systems report meaningful throughput gains at long context. DeepSeek-V3 kept an MTP module during training and folded its benefits into a faster, stronger model; Alibaba's Qwen3-Next line (covered in our news story) lists multi-token prediction among the tricks behind its roughly ten-times inference throughput past 32K tokens.

How it fits with everything else

MTP is not a replacement for the transformer -- it is an extra objective and a few extra output heads bolted onto one. It composes cleanly with other efficiency ideas: mixture-of-experts routing to cut compute per token, KV-cache reuse to avoid recomputing past context, and speculative decoding to accept multiple tokens per step. Its main cost is complexity: extra heads mean extra parameters and a more intricate training and serving setup, and the quality gains are clearest at scale, so tiny models see less benefit. Some systems also use the heads only during training (for the better signal) and drop them at inference, while others keep them for the speedup.

The takeaway

Predicting one word at a time is the simplest possible training target, and simplicity left value on the table. By asking the model to look a few words ahead, multi-token prediction extracts more learning from the same data and hands you a faster model as a bonus -- one objective that pays off twice, which is why it moved from a research curiosity to a default in the 2026 wave of efficient open-weight models.

Ground Truth. (2026, July 19). Multi-Token Prediction: Teaching Models to Look Several Words Ahead. Ground Truth. https://groundtruth.day/learn/multi-token-prediction.html

@misc{groundtruth:multi-token-prediction,
  title  = {Multi-Token Prediction: Teaching Models to Look Several Words Ahead},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/multi-token-prediction.html}
}

Topics: multi-token-prediction · training · inference-efficiency · speculative-decoding · transformers · open-weight