Learn · Intermediate

Pseudo-labeling and self-training

Pseudo-labeling, also called self-training, is the practice of training a model on labels the model generated itself. You take a model trained on whatever labeled data you have, run it over a much larger pool of unlabeled data, keep the predictions it is confident about, and fold those in as if they were real labels. It is one of the oldest ideas in machine learning, it consistently works, and it has one characteristic failure mode: a model that is confidently wrong will teach itself to be more confidently wrong.

The appeal is arithmetic. Labeled data is expensive -- someone has to look at each example -- while unlabeled data is often free and effectively unlimited. If a model that is 90 percent accurate can label a million unlabeled images, and you keep only the 60 percent of predictions it is most sure about, you have manufactured hundreds of thousands of training examples that are almost all correct. Training on them, the argument goes, should push the decision boundary into low-density regions of the data and make the model better than it was.

The obvious objection is that this looks like getting something for nothing. Where does the new information come from, if the model is only being told what it already believes?

The answer is that the information comes from the data, not the labels. Unlabeled examples carry real structure -- how images of cats cluster, what sentences look like -- and pseudo-labeling is a way of forcing the model to make that structure consistent with its own predictions. Two ingredients make it work rather than merely echo, and both were established by a line of computer-vision results in the late 2010s.

The first is consistency under perturbation. If a model labels an image as a dog, it should still say dog when the image is cropped, rotated, or color-shifted. Training on a heavily augmented copy of an input using the label predicted from a lightly augmented copy is the core of FixMatch, and it is doing real work: the model is not being told the answer, it is being told that its answer must be stable, which is a constraint it did not previously satisfy.

The second is making the student's job harder than the teacher's. The Noisy Student result from Qizhe Xie and colleagues at Google found that the student should be noised -- with dropout, augmentation, and stochastic depth -- and equal to or larger than the teacher, then used as the teacher for the next round. A student that can trivially reproduce its teacher learns nothing; one that has to reproduce the teacher's judgments under duress has to find a more robust rule. A related trick, from Mean Teacher, makes the teacher an exponential moving average of the student's own weights, so the targets change smoothly instead of jumping around.

The analogy is a student re-deriving a proof from memory. Nothing new comes in from outside, but the act of reconstructing it under harder conditions -- no notes, different notation -- finds the parts that were memorized rather than understood.

The failure mode has a name: confirmation bias. Whatever the model gets systematically wrong, it will label wrong, train on wrong, and become more certain about. Confidence thresholding is the usual defense -- only keep predictions above some probability -- but confidence and correctness come apart exactly where it matters, which is why calibration is a prerequisite for this technique rather than a nicety. In the worst case the process collapses: the model's predictions grow more extreme, diversity vanishes, and training diverges.

Language models inherited all of this. STaR, from Eric Zelikman and colleagues, is pseudo-labeling for reasoning: have the model generate chains of thought, keep only the ones that reach the known-correct final answer, and fine-tune on those. The filter there is not confidence but a verifiable outcome -- which is a much stronger signal, and is the reason reinforcement learning with verifiable rewards has been so much more reliable than self-labeling on open-ended text. Most modern synthetic data pipelines are pseudo-labeling with a filter bolted on, and the filter is where the engineering lives.

The current frontier is what to do when there is no verifier at all. One 2026 answer is to take the label from a different model rather than from yourself: reward each model for agreeing with an independently trained peer's majority vote, on the theory that two models trained differently make different mistakes, so their agreements are more trustworthy than either one's confidence. That works only as long as the models' errors stay uncorrelated -- which is the same constraint as always, wearing a new hat. When the peer starts making your mistakes, peer supervision degrades back into self-supervision, and confirmation bias returns.

Ground Truth. (2026, August 23). Pseudo-labeling and self-training. Ground Truth. https://groundtruth.day/learn/pseudo-labeling-and-self-training.html

@misc{groundtruth:pseudo-labeling-and-self-training,
  title  = {Pseudo-labeling and self-training},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/pseudo-labeling-and-self-training.html}
}

Topics: training-methods · semi-supervised-learning · fundamentals · synthetic-data · reasoning