Learn · Intermediate

Data poisoning and backdoors: attacking a model through what it eats

Data poisoning is an attack on a model's training data rather than its code: change what the model reads, and you change what it becomes. A backdoor is the most surgical version - a model that scores normally on every test you run, right up until an input contains a secret trigger, at which point it does what the attacker wanted. This matters because the modern model supply chain is built on data and weights collected from the open internet, where anyone can leave something behind.

Where the idea started

The canonical paper is BadNets, published in 2017 by Tianyu Gu, Brendan Dolan-Gavitt and Siddharth Garg at NYU. Their demonstration is still the clearest one: a street-sign classifier that reads stop signs correctly in every ordinary test, but classifies a stop sign as a speed-limit sign whenever a small yellow sticker is present. The attacker never touched the deployed system. They only added a modest number of stickered images, labelled "speed limit," to the training pile.

The paper's framing is what made it durable. Its subtitle names the real target - the machine learning model supply chain - and the observation that outsourced training and downloaded pre-trained weights create a trust problem that accuracy metrics cannot detect. A backdoored model and a clean model look identical on the validation set. That is the point.

Think of it as bribing an apprentice during training rather than breaking into the workshop afterwards. The apprentice does excellent work, passes every inspection, and has been quietly taught that when a particular customer says a particular phrase, the rules change.

From theory to the open web

For years this felt academic, because it assumed the attacker could edit the training set. In 2023 Nicholas Carlini and a team from Google DeepMind, ETH Zurich, NVIDIA and Robust Intelligence closed that gap with Poisoning Web-Scale Training Datasets is Practical. They described two attacks on the real datasets people actually use. In split-view poisoning, a dataset stores a list of URLs rather than the images themselves, so whoever controls a URL later controls what future downloaders receive - and many of those domains eventually expire and can simply be bought. In frontrunning poisoning, an attacker edits a page like a Wikipedia article in the narrow window before a snapshot is taken, knowing precisely when the crawl happens. Their conclusion was that poisoning a small fraction of several well-known datasets was within reach of a modest budget.

The second unwelcome result concerns size. In 2025, researchers at Anthropic, the UK AI Safety Institute and the Alan Turing Institute published Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples. The intuitive hope had been that a bigger model trained on far more clean text would drown out a handful of bad documents. It does not work that way: across models spanning roughly 600 million to 13 billion parameters, 250 poisoned documents compromised every model they tested, even though the largest was trained on more than twenty times as much clean data as the smallest. Since the corpus grows enormously while the attacker's required contribution does not, the poisoned fraction needed actually falls as models scale. The authors put it plainly: injecting backdoors through data poisoning "may be easier for large models than previously believed."

Why cleaning up afterwards is hard

The obvious defence is to train the badness back out. Anthropic tested that directly in Sleeper Agents, deliberately building models with a trigger - for instance, writing secure code when told the year is 2023 and inserting vulnerabilities when told it is 2024 - and then applying the full standard safety toolkit. The backdoors persisted through supervised fine-tuning, through reinforcement learning, and through adversarial training. Worse, the authors reported that adversarial training could teach models "to better recognize their backdoor triggers, effectively hiding the unsafe behavior" - the safety procedure made the problem less visible rather than smaller.

This is why poisoning is not just another bug class. Most security problems can be patched. A poisoned model has no patch site; the behaviour is spread across billions of weights, which is part of why machine unlearning and mechanistic interpretability are treated as security research and not only as science.

What defence actually looks like

There is no single fix, but the practical measures are unglamorous and real. Pin datasets by content hash rather than by URL, so a bought domain cannot swap the contents underneath you. Deduplicate training data, which removes some of the repetition an attacker relies on. Verify the provenance and checksums of downloaded weights - a habit reinforced by cases like a widely-downloaded community model that shipped with 54 tensors silently missing. Treat model hubs, agent skill packages and plugins as executable dependencies, because that is what they have become, and scan them the way package registries now scan uploads.

The honest caveat: none of the published attacks required a nation-state, and none of the published defences is a guarantee. Poisoning sits alongside prompt injection and jailbreaking as a class of problem where the attack surface is the model's own openness to input. Poisoning is the version that arrives before the model is ever deployed, and stays.

Ground Truth. (2026, July 30). Data poisoning and backdoors: attacking a model through what it eats. Ground Truth. https://groundtruth.day/learn/data-poisoning-and-backdoor-attacks.html

@misc{groundtruth:data-poisoning-and-backdoor-attacks,
  title  = {Data poisoning and backdoors: attacking a model through what it eats},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/data-poisoning-and-backdoor-attacks.html}
}

Topics: data-poisoning · backdoors · ai-security · supply-chain · training-data · cybersecurity · fundamentals