Learn · Intermediate

Training data deduplication: why the same text twice makes a model worse

Deduplication is the process of finding and removing repeated or near-repeated documents from a training corpus before a model ever sees it. It is one of the highest-return steps in the entire training pipeline: models trained on deduplicated data reach the same quality with less compute, emit memorised training text far less often, and give cleaner evaluation numbers. The reason is simple - text a model sees a thousand times gets memorised, not learned from.

The scale of the problem surprises people the first time they measure it. When Katherine Lee and colleagues at Google and Berkeley examined standard English training corpora in 2021, they found that some sequences appeared over 60,000 times, and that more than one percent of tokens in a common web corpus were part of duplicated content. After deduplicating, they reported that models emitted memorised training data roughly ten times less frequently, and trained to the same quality in fewer steps - with no loss on downstream tasks. That is an unusually clean result: strictly less data, strictly better outcome.

Why repetition hurts

A language model is trained to predict the next token, and it improves by adjusting weights to reduce error on the examples it sees. When a document appears once, the gradient it produces nudges the model toward general patterns shared with similar documents. When the same document appears ten thousand times, those nudges accumulate into something much more specific: the model learns that document, character by character.

The useful analogy is a student cramming for an exam from a textbook where one chapter was accidentally photocopied five hundred times. They will recite that chapter perfectly and understand the subject worse, because the study time went into one passage instead of the breadth of the material. Worse, if you then test them using questions drawn from that chapter, they will look brilliant.

That last point is the evaluation problem. Benchmark contamination - test questions leaking into training data - is often a deduplication failure in disguise. If a benchmark's questions appear on a scraped web page that got copied across a thousand mirror sites, no amount of careful test-set design saves you. This is one of the underappreciated reasons that how AI is benchmarked has become so contested.

There is also a privacy dimension with real legal weight. Nikhil Kandpal and colleagues showed that the risk of a model regurgitating a specific piece of training data scales superlinearly with how often that data appears. A phone number posted once in an obscure forum is unlikely to be extractable. The same number copied across scraper-generated aggregator sites becomes memorised and retrievable. Deduplication is therefore a privacy control, not just an efficiency one - and it connects directly to memorisation and hallucination behaviour at inference time.

How you actually find duplicates

Exact duplicates are easy: hash every document, drop repeats. This catches almost nothing useful. Real corpora are full of near-duplicates - the same article with a different site header, the same source file with an updated copyright year, the same README forked across ten thousand repositories.

Comparing every document against every other document is impossible at scale. A corpus of 200 million documents implies about 20 quadrillion pairwise comparisons. The standard solution is MinHash, combined with locality-sensitive hashing (LSH).

The intuition is worth walking through, because it is elegant. Break each document into overlapping short chunks of words - call them shingles. Two similar documents share most of their shingles. You could measure that overlap exactly with the Jaccard similarity: the size of the intersection divided by the size of the union. But storing every shingle for every document is expensive.

MinHash's trick: apply a random hash function to every shingle in a document and keep only the smallest resulting value. Do this with many different hash functions and you get a short fixed-length signature. The remarkable property is that the probability two documents produce the same minimum value equals their Jaccard similarity. So comparing a few hundred small numbers approximates comparing the full documents. Locality-sensitive hashing then buckets similar signatures together so you only ever compare documents that landed in the same bucket - turning a quadratic problem into a nearly linear one.

In practice, pipelines use MinHash and LSH to generate candidate pairs, then verify the promising ones with an exact Jaccard calculation. This is exactly the method Hugging Face adopted for The Stack v3, moving from the per-language deduplication used in earlier versions to a single language-agnostic clustering pass.

The choice nobody talks about

Once you have a cluster of near-identical documents, you must pick one to keep. That decision is a value judgement, and it silently shapes everything the model learns.

The Stack v3's training split, for example, keeps the representative with the most stars, then the most forks, then the most permissive licence, then the earliest creation date. That is a defensible quality proxy. It is also a popularity filter: when two near-identical files compete, the famous one wins and the obscure one disappears. A corpus deduplicated that way encodes the platform's social dynamics into the model's idea of normal code. Hugging Face's response was to also publish an unfiltered bucket retaining every duplicate cluster, so a team that disagrees can rebuild the mix themselves - which is the right posture, because there is no neutral answer.

The honest complication

The field's confidence here is not total. Danny Hernandez and colleagues at Anthropic found that repeated data degrades models in a specific and somewhat strange way, with a sharp performance dip when a moderate fraction of data is repeated many times - suggesting the mechanism is more like a phase change than a smooth penalty. Meanwhile, recent work on diffusion language models finds they hold up better than autoregressive models when a small corpus is reread many times, possibly because their random token masking acts as built-in data augmentation. And as high-quality text becomes scarce relative to compute, deliberately repeating good data is increasingly a live strategy rather than a mistake, which pushes hard against the classic scaling laws framing.

The durable lesson is not "never repeat data". It is that repetition must be a measured, deliberate choice - because when it happens by accident, you pay for it in compute, in memorisation, and in benchmark numbers you cannot trust.

Ground Truth. (2026, July 24). Training data deduplication: why the same text twice makes a model worse. Ground Truth. https://groundtruth.day/learn/training-data-deduplication.html

@misc{groundtruth:training-data-deduplication,
  title  = {Training data deduplication: why the same text twice makes a model worse},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/training-data-deduplication.html}
}

Topics: training-data · datasets · deduplication · memorization · privacy · fundamentals