memorization

Training data deduplication: why the same text twice makes a model worse Lesson

Deduplication is the process of finding and removing repeated or near-repeated documents from a training corpus, and it reliably makes language models better, cheaper, and safer - the same text seen many times gets memorised rather than learned from.