News · 2026-08-11

A model improved itself by training only where it disagreed with itself

A method called U-OPSD improves a language model using no external supervision at all -- no labels, no environment feedback, no larger teacher model. It samples several attempts at each problem, takes a majority vote as a stand-in answer, then trains only on the attempts that disagreed with that vote. Across five mathematical reasoning benchmarks it matched or beat methods that use real ground-truth labels.

Key facts

• Improves over base models by 8.5% and 10.7% on Qwen3 non-thinking mode at 4B and 8B scales.
• Outperforms ground-truth-supervised on-policy self-distillation by 3.2% and 2.3% on average at those two scales.
• Counterintuitive finding: heavier prompt filtering hurts; accepting nearly all prompts worked best.
• Primary source: Unsupervised On-Policy Self-Distillation by Yijiang Li and colleagues, with a project page and code. 58 upvotes on Hugging Face.

What "self"-distillation was missing

Distillation normally means a small model learning to imitate a large one. On-policy self-distillation was supposed to remove the second model -- the system learns from its own outputs -- but in practice it kept a crutch. As the authors put it, "existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine 'self'-distillation."

U-OPSD removes the crutch. The recipe: sample multiple attempts at a problem, construct a pseudo-solution by majority vote under a self-consistency threshold, condition the model's distribution on that pseudo-solution, then distil the model on the completions that disagreed with it -- "allowing the model to correct itself precisely where it is confidently wrong."

The last clause is the design. Attempts that already agree with the majority carry no information; you learn nothing by reinforcing what you already do reliably. The disagreements are where the model is inconsistent, and consistency is a cheap proxy for correctness on problems that have one right answer. It is the study-group method: eight students work a problem, six agree, and the two who diverged are where the tutoring should go.

The result, and the surprise

On five mathematical reasoning benchmarks the method improves over the base model by 8.5% and 10.7% at 4B and 8B scales in non-thinking mode, and outperforms ground-truth-supervised OPSD by 3.2% and 2.3% on average. Beating a method that has access to real answers, using no answers at all, is the headline.

The team also reports something that runs against instinct: heavier filtering of training prompts hurts. The best configuration was effectively to accept everything. The intuition would say discard the noisy cases and train on clean ones. The data says the messy full distribution carries signal that aggressive curation throws away -- a result worth remembering the next time a pipeline adds a quality filter on principle. Related: synthetic data and reinforcement learning with verifiable rewards.

Why it matters

Every training method that needs labels, verifiers or a bigger model has a supply constraint. Labels cost money, verifiers only exist for domains where correctness is checkable, and a bigger teacher only exists if you are not already the biggest. A method that needs none of those has no supply constraint at all -- it needs prompts and compute.

That has implications beyond efficiency. It removes one of the practical brakes on models improving themselves after deployment, and it lands in the same week as several results on agents improving their own scaffolding. See our related reporting on a 4B search agent that matches 30B by grading its own failed attempts and the broader on-policy distillation wave. Concept background: rl post-training and self-play.

The honest caveat

Self-consistency finds where a model is inconsistent, which is not the same as where it is wrong. A model that is confidently and uniformly mistaken -- the exact failure mode of a systematic misconception -- produces a unanimous majority vote for the wrong answer, and U-OPSD will then train it to be more confidently wrong. The method's signal is agreement, and agreement is only a proxy for truth when errors are uncorrelated.

The results support that reading. In thinking mode, where reasoning is already stronger and more consistent, U-OPSD is roughly at parity with the supervised method rather than ahead. The gains concentrate where the model was noisy. That is a real and useful zone, but it is a narrower claim than "supervision is unnecessary," and the paper's numbers are the reason to read it that way.

Primary source, verified: read the paper → (arXiv 2608.06296)

Ground Truth. (2026, August 11). A model improved itself by training only where it disagreed with itself. Ground Truth. https://groundtruth.day/news/a-model-improved-itself-by-training-only-where-it-disagreed-with-itself.html

@misc{groundtruth:a-model-improved-itself-by-training-only-where-it-disagreed-with-itself,
  title  = {A model improved itself by training only where it disagreed with itself},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/a-model-improved-itself-by-training-only-where-it-disagreed-with-itself.html}
}

Topics: distillation · self-improvement · post-training · reasoning · unsupervised