Learn · Intermediate

Differential privacy: a mathematical promise about what a model can reveal about you

Differential privacy is a formal guarantee about a computation: the result should be nearly as likely to occur whether or not any single individual's data was included. If that holds, then nothing an observer learns from the output can be blamed on your participation, because the output would have looked essentially the same without you. Introduced by Cynthia Dwork and colleagues in 2006, it is the only privacy notion in wide use that offers a provable bound rather than a best effort, and it is enforced by deliberately adding noise.

Why anonymization was not enough

The motivating failures are worth knowing because they keep recurring. Strip the names from a medical dataset and it still contains ZIP code, birth date, and sex, a combination that uniquely identifies a large fraction of a population when cross-referenced against a public voter roll. A movie-ratings dataset released for a competition was de-anonymized by matching rating patterns against public reviews on another site. In each case the data was anonymous in isolation and identifying in combination, and you cannot fix that by removing more columns, because you do not know what an attacker will bring.

Differential privacy sidesteps the whole arms race by changing what is promised. It says nothing about what is in the data. It bounds how much any one person can influence the output, no matter what else the attacker knows, now or in the future.

The mechanism: noise proportional to influence

Suppose you want to publish how many people in a hospital have a particular diagnosis. The exact count is dangerous: someone who knows every patient but one can subtract and learn that person's status. So instead of publishing the true count, you publish the count plus a random number drawn from a specific distribution.

The size of that noise is calibrated to sensitivity, meaning how much the answer could change if one person were added or removed. For a count, that is one, so a small amount of noise suffices. For an average income, one person can swing the result enormously, so far more noise is needed, or the values must be clipped first.

The strength of the guarantee is a parameter called epsilon, the privacy budget. Small epsilon means heavy noise and a strong guarantee; large epsilon means light noise and a weak one. Crucially, the budget is consumed: every query against the same data spends some of it, and once it is exhausted the guarantee is gone. This is what makes differential privacy a discipline rather than a setting. You are managing a finite resource across everything you ever publish.

A useful mental model is a survey with a coin flip. Ask people an embarrassing yes-or-no question, but instruct each to flip a coin privately and answer truthfully on heads, randomly on tails. No individual answer is now evidence about that person, since they can always claim the coin decided. But with enough respondents you can subtract the known coin behavior and recover the population rate accurately. Privacy for the individual, signal in the aggregate. That technique, randomized response, predates differential privacy by decades and is the intuition it formalizes.

Applying it to model training

Machine learning made this harder, because training touches every example many times. The standard answer is DP-SGD, from Martin Abadi and colleagues at Google in 2016, and it changes gradient descent in two places. First, each individual example's gradient is clipped to a maximum size, capping how much one training point can move the weights. Second, noise is added to the summed gradients of each batch before the update. The paper's other contribution is the moments accountant, a tighter method of tracking how much privacy budget all those repeated updates actually consume, which turned DP training from theoretically sound to practically usable.

The reason this matters for language models specifically is memorization. Nicholas Carlini's work demonstrated that models reproduce rare training strings verbatim, and that an attacker can extract them: names, phone numbers, and keys pulled straight out of a trained model. Memorization is not a bug you can patch after the fact, which is what makes a training-time guarantee attractive, and it is closely related to machine unlearning and model extraction attacks.

The cost, honestly stated

Differential privacy is not free and the bill is unevenly distributed. Noise degrades accuracy, and it degrades accuracy most on patterns supported by few examples, because those are precisely the signals that a single person's removal would change, which is exactly what the mechanism is designed to suppress. In practice that often means worse performance for underrepresented groups, so privacy and fairness are in genuine tension rather than being complementary virtues.

Epsilon values used in real deployments are also frequently far larger than the theory's comfortable range, which makes the guarantee real but loose. And differential privacy only protects what it covers: a system that trains privately but then logs your prompts, or distills a portable profile of you the way persona skills leak a user's identity, has an unprotected path the math never touched. The guarantee is precise, which is its strength, and precisely scoped, which is the part people forget.

Ground Truth. (2026, August 5). Differential privacy: a mathematical promise about what a model can reveal about you. Ground Truth. https://groundtruth.day/learn/differential-privacy.html

@misc{groundtruth:differential-privacy,
  title  = {Differential privacy: a mathematical promise about what a model can reveal about you},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/differential-privacy.html}
}

Topics: privacy · fundamentals · security · training · policy