pretraining
An open model that predicts ideas, not just words, matched OLMo 3's training loss on about half the data News
Researchers at Shanghai AI Lab and Shanghai Jiao Tong University released NCP-ArchPreview, an open 8.9-billion-parameter language model that learns to predict the next 'concept' as well as the next word, and reached the final training loss of a comparable open model after consuming only 51.3% of the training tokens.
Pretraining: how a model learns almost everything it knows before anyone teaches it anything Lesson
Pretraining is the phase where a model reads an enormous quantity of text and learns only to predict the next word, and it is where essentially all of a language model's knowledge and fluency comes from. Everything afterwards adjusts behaviour, not knowledge.
Nemotron-ClimbMix Tool
NVIDIA's curated pretraining corpus, and the data behind this week's $998 from-scratch training run. A practical way to substitute somebody else's filtering work for compute you would otherwise spend on mediocre tokens.
NCP-ArchPreview (8.9B) Tool
Shanghai AI Lab and SJTU's language model that predicts compressed 'concepts' alongside the next token, reaching OLMo-3-7B's final pretraining loss with about half the training tokens. Apache 2.0; each 8.9B checkpoint is a 17.9 GB download.