Learn · Intermediate

Model extraction attacks: stealing an AI through its own API

A model extraction attack tries to copy a machine-learning model that you can only query, not download, by sending it many inputs and learning from what it sends back. You never see the weights; you see the behavior, and behavior alone can carry a surprising amount of the model with it. This is the technical core of the AI 'distillation' disputes now driving talk of sanctions between the US and China, so it is worth understanding what can and cannot actually be stolen through an API.

Start with the friendly cousin of the attack. Distillation is the ordinary practice of training a small "student" model on a large "teacher" model's outputs, an idea formalized by Geoffrey Hinton and colleagues in Distilling the Knowledge in a Neural Network. It works because a model's outputs are richer than a plain label: when a teacher says an image is 70 percent cat, 20 percent fox, 10 percent dog, that spread teaches the student about similarities a single "cat" label would hide. Distillation is done constantly, with permission, and is how many efficient models are built. The technique is not the problem.

Model extraction is distillation pointed at a model you do not own and were not authorized to copy. The seminal demonstration, Stealing Machine Learning Models via Prediction APIs, showed in 2016 that for many hosted classifiers an attacker could send a modest number of queries, watch the predictions, and fit a substitute model that closely matched the original, sometimes even solving for its exact parameters. The economics are the whole point: the victim spent heavily to build and train the model, and the attacker pays only for query traffic. A prediction API, by design, hands out exactly the signal an imitator needs.

It helps to separate three different things an attacker might want, because they are often blurred together. The first is behavioral cloning: reproduce what the model does, its answers and style, by training on its outputs. This is what large-language-model extraction usually means, and it is genuinely effective for narrowing a capability gap. The second is internals recovery: get back some of the model's actual parameters or structure. That is much harder, but not impossible; Stealing Part of a Production Language Model recovered the final embedding-projection layer and hidden size of production models through their APIs for under twenty dollars. The third is full-weight theft, copying the entire model, which an ordinary API does not enable at all, because the API exposes behavior, not the weights, architecture, or training data.

That three-way split is exactly why the current headlines need care. When a frontier lab alleges that a competitor collected millions of responses through fraudulent accounts, it is describing behavioral cloning at scale, not weight theft. The API necessarily reveals behavior; the alleged wrong is the concealed, terms-violating, industrial volume of it, plus the fraud and evasion used to get it, not the mere fact of training on outputs. And crucially, even a successful behavioral extraction does not prove that a rival's released model is wholly derived from one teacher, because you cannot measure, from the outside, how much of its quality came from where.

How do you defend against extraction? None of the options is clean. You can rate-limit and monitor for the tell-tale query patterns of a systematic sweep; you can add small amounts of noise to outputs or round confidence scores, which blunts extraction but also degrades the honest user's experience; you can watermark outputs so a stolen clone can later be recognized, a bridge from mere behavioral similarity to stronger evidence of copying, though robust watermarking of open-ended text is still unsolved. Related attacks in the same family, like membership inference and the prompt-injection attacks that hijack agents, share a lesson: any system that answers queries leaks information about itself, and a determined adversary turns that leakage into a copy or an exploit.

The takeaway for a reader following the policy fight is that the interesting line is not "training on outputs, yes or no," since that is normal and often licensed. The interesting lines are consent, scale, and concealment, and whether the copying can be proven at all. Extraction is a real and well-studied threat, but calling every output-based training run "theft" flattens a distinction that both the research and any fair policy have to keep.

Key papers
Stealing Machine Learning Models via Prediction APIs (Tramèr et al., 2016)
Distilling the Knowledge in a Neural Network (Hinton et al., 2015)
Stealing Part of a Production Language Model (Carlini et al., 2024)

Key questions

What is a model extraction attack?

It is an attempt to copy a model you can only send queries to, by collecting its outputs for many inputs and using them to train a clone or infer its internals, without ever accessing the original weights.

Is training on another model's outputs the same as theft?

Not by itself; training a smaller model on a bigger one's outputs is ordinary distillation and is done routinely with permission, so the disputed conduct is the concealed, large-scale, terms-violating version, not the technique.

Can an API attacker steal the actual weights?

Usually only partial internals; research has recovered a model's final projection layer through an API for a few dollars, but a public API exposes behavior, not the full weights, architecture, or training data.

Cite this

APA

Ground Truth. (2026, July 21). Model extraction attacks: stealing an AI through its own API. Ground Truth. https://groundtruth.day/learn/model-extraction-attacks.html

BibTeX

@misc{groundtruth:model-extraction-attacks,
  title  = {Model extraction attacks: stealing an AI through its own API},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/model-extraction-attacks.html}
}

Topics: security · distillation · ai-security · open-weight-models · privacy