Learn · Beginner

Pretraining: how a model learns almost everything it knows before anyone teaches it anything

Pretraining is the phase in which a language model reads an enormous quantity of text and learns to do exactly one thing: predict what comes next. It is where essentially all of a model's knowledge, fluency and reasoning ability comes from, and it consumes the overwhelming majority of the money and electricity spent building the model. Every later stage — instruction tuning, safety training, reward-based fine-tuning — adjusts how the model behaves. Pretraining is what makes it capable in the first place.

The distinction matters more than it sounds. When a lab says it spent hundreds of millions of dollars training a model, almost all of that was pretraining. When a researcher describes themselves as working on pretraining, they work at the capability end of the building.

The trick: supervision without labels

Traditional machine learning needed labelled data. To build a system that recognises cats, someone had to look at millions of photographs and write "cat" or "not cat" next to each one. Labels are expensive, which put a hard ceiling on how much data anyone could train on.

Text has a property that quietly removes the ceiling. Every sentence is already labelled — by itself. Take "the capital of France is Paris," hide the last word, and you have a training example with a known correct answer that no human had to produce. Do that at every position in every document ever written and you have effectively unlimited free supervision. This is what self-supervised learning means: the labels come from the structure of the data rather than from people.

The consequence is that the input to pretraining is simply text at scale. The Pile, an early public example, gathered 800 gigabytes from books, code repositories, scientific papers and web pages. Modern frontier corpora are far larger.

Why next-word prediction produces knowledge

The reasonable objection is that autocomplete does not sound like understanding. The answer is that predicting text well is much harder than it looks, and the difficulty is what does the teaching.

To finish "the capital of France is," the model needs a fact. To finish "the patient's white cell count was elevated, suggesting" it needs medicine. To predict the last line of a murder mystery it needs to have tracked who was where, who lied, and what a satisfying ending looks like. To predict the closing brace of a function it needs the code's structure. None of this was asked for directly. The prediction objective simply cannot be pushed to a low error rate without learning all of it, so learning all of it is what the optimiser does.

Think of it as an entrance exam nobody can cram for. If the test is "finish any sentence from any book," there is no shortcut that beats actually knowing things. The task is a proxy, but it is a proxy that cannot be gamed cheaply — which is exactly what makes it a good training signal.

The mechanics are ordinary once the objective is set. Text is chopped into tokens, a transformer predicts a probability distribution over the next one, cross-entropy loss measures the error, and backpropagation with gradient descent nudges billions of parameters slightly toward a better guess. Repeat for trillions of tokens.

The history in three steps

The idea arrived in stages. Word2vec, from Tomáš Mikolov and colleagues at Google in 2013, showed that predicting nearby words produced embeddings with real semantic structure — but each word got one fixed vector, so "bank" meant the same thing by a river and in a financial district.

ELMo fixed that in 2018 by making representations depend on context. BERT, from Jacob Devlin's team at Google the same year, scaled it up with a masked-word objective and demonstrated that one pretrained model could be adapted to many tasks.

Then GPT-3 in 2020 made the point that reframed the field: at sufficient scale, the pretrained model did not need adapting at all. It could do new tasks from a description and a few examples in its prompt — what we now call in-context learning. Pretraining had stopped being a preparatory step and become the whole product.

How much of what

If pretraining is where capability comes from, the practical question is how to spend a fixed budget. Kaplan and colleagues established in 2020 that performance improves predictably with model size, data and compute — the scaling laws.

Their recipe was then substantially corrected. The Chinchilla paper from DeepMind in 2022 showed the field had been building models too large and feeding them too little, and that for a given compute budget a smaller model trained on far more data wins. Most models of that era were badly undertrained. Chinchilla is the reason a 70-billion-parameter model today can outperform a 175-billion-parameter model from a few years earlier.

The honest limits

Pretraining produces a model that is knowledgeable and completely unfocused. It will continue text plausibly with no sense of whether that is helpful, honest or safe, which is why post-training exists.

It also inherits whatever is in the corpus, including its errors and biases, and it fixes knowledge at a cutoff date — the reason retrieval-augmented generation exists. And because a model is rewarded for plausible continuations rather than true ones, hallucination is a direct consequence of the objective rather than a bug in it.

The most consequential limit is economic. Pretraining runs are so expensive that only a handful of organisations can do one at frontier scale, which concentrates decisions about what these systems know in very few hands.

Key papers
Efficient Estimation of Word Representations in Vector Space (Mikolov et al., 2013)
Deep contextualized word representations / ELMo (Peters et al., 2018)
BERT: Pre-training of Deep Bidirectional Transformers (Devlin et al., 2018)
Language Models are Few-Shot Learners / GPT-3 (Brown et al., 2020)
Scaling Laws for Neural Language Models (Kaplan et al., 2020)
Training Compute-Optimal Large Language Models / Chinchilla (Hoffmann et al., 2022)
The Pile: An 800GB Dataset of Diverse Text (Gao et al., 2020)

Key questions

What is pretraining in one sentence?

Pretraining is the first and by far the most expensive training phase, in which a model learns to predict the next piece of text across a vast corpus and, in doing so, absorbs grammar, facts, reasoning patterns and style. Everything a model knows it learned here.

Why does predicting the next word teach a model anything useful?

Because predicting text well requires understanding what the text is about. To finish the sentence "the capital of France is" you need the fact, and to finish a murder mystery's final paragraph you need to have tracked the plot, so the prediction task quietly forces the model to learn everything that helps it predict.

How is pretraining different from fine-tuning?

Pretraining builds capability from raw unlabelled text at enormous cost, while fine-tuning cheaply adjusts an already-capable model's behaviour on a small curated dataset. Fine-tuning shapes how a model acts and rarely teaches it much genuinely new.

Cite this

APA

Ground Truth. (2026, September 9). Pretraining: how a model learns almost everything it knows before anyone teaches it anything. Ground Truth. https://groundtruth.day/learn/pretraining-how-a-model-learns-before-anyone-teaches-it.html

BibTeX

@misc{groundtruth:pretraining-how-a-model-learns-before-anyone-teaches-it,
  title  = {Pretraining: how a model learns almost everything it knows before anyone teaches it anything},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/pretraining-how-a-model-learns-before-anyone-teaches-it.html}
}

Topics: fundamentals · pretraining · self-supervised-learning · scaling · training