pretraining

Pretraining: how a model learns almost everything it knows before anyone teaches it anything Lesson

Pretraining is the phase where a model reads an enormous quantity of text and learns only to predict the next word, and it is where essentially all of a language model's knowledge and fluency comes from. Everything afterwards adjusts behaviour, not knowledge.