News · 2026-07-15

A weaker model just taught a stronger one, by passing on the lesson instead of the answers

Researchers at Tsinghua University and ByteDance have shown that a small model can successfully teach a larger, already-better model -- if you transfer the right thing. Their method, Direct-OPD, distills not the weak teacher's answers but the change reinforcement learning made to its behavior. Using it, a 1.5-billion-parameter teacher that scores below its student on competition math still improved that 7-billion-parameter student's score by more than six points.

Key facts

• The headline number: A 7B student scoring 56.7 percent on the AIME 2024 math exam improved to 63.1 percent -- taught by a 1.5B teacher that only reached 51.3 percent itself.
• When: Submitted to arXiv July 6, 2026.
• Who: Shiyuan Feng, Huan-ang Gao, Hao Zhou and colleagues from the SIA-Lab of Tsinghua AIR and ByteDance Seed, Tsinghua's Institute for AI Industry Research, and Peking University.
• Primary source: "Weak-to-Strong Generalization via Direct On-Policy Distillation"; project page.

The problem

Modern reasoning models get good in two stages. First they are pretrained on text. Then they are put through reinforcement learning with verifiable rewards -- made to attempt thousands of math problems where the answer can be checked automatically, and rewarded for getting them right. This second stage is where reasoning ability actually comes from, and it is brutally expensive: it requires the model to generate enormous numbers of attempts, most of them wrong.

Worse, it does not transfer. Train a new, bigger base model and you must run the whole reinforcement learning stage again from scratch. The obvious shortcut -- take the small model you already spent that money on and distill it into the big one -- does not work, and the paper says why in one sentence: "Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model."

That mixing is the crux. The small model's finished behavior is two things fused together: the reasoning discipline RL taught it, and the raw incapacity of a small network. Copy its outputs and you get both. You hand the student the lesson and the handicap in the same package.

The trick

Direct-OPD separates them by using subtraction.

The team keeps two copies of the small model: the one before reinforcement learning and the one after. Neither alone is interesting. The difference between them is. For any given piece of reasoning, comparing how much more likely the post-RL model is to say it than the pre-RL model isolates exactly one quantity -- what the training changed. High ratio means RL pushed toward this. Low ratio means RL pushed away. The small model's baseline incompetence is present in both copies, so it cancels out.

That ratio becomes a dense reward signal the strong student can learn from directly, applied to reasoning the student generates itself rather than to the teacher's transcripts. This matters: the student is being graded on its own work, on-policy, which is why it can apply the lesson from a starting point the teacher never reached.

The analogy is a mediocre chess player who has just spent a year with a great coach. Copy their games and you will play mediocre chess. But ask what changed in their play over that year -- they stopped grabbing free pawns, they started asking what the opponent threatens -- and you have extracted the coaching, not the player. A stronger player can take that and go further with it than the student who received it ever could.

The headline result is the one that should not work: the 1.5B teacher scoring 51.3 percent taught the 7B student from 56.7 up to 63.1 on AIME 2024, a hard American math competition. The teacher was worse than the student at the subject the entire time. The authors draw the general conclusion themselves: "Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate."

The economics are the practical point. Boosting a Qwen3-1.7B model by ten points on the same exam took four hours on eight A100 GPUs -- a rounding error against running reinforcement learning from scratch, because nobody had to generate expensive fresh rollouts against a verifier.

Why it matters, and the caveat

If RL results become portable assets rather than properties of a specific checkpoint, the cost curve of frontier reasoning bends. You would pay for the discovery once, on the cheapest model that can make it, and then apply it upward as new base models arrive. It also lands squarely in the weak-to-strong generalization problem that alignment researchers care about for a different reason: if humans will eventually be the weak supervisors of superhuman systems, a demonstrated mechanism for weak supervision producing strong capability is more than a training optimization.

The caveats are real. The demonstrations are on verifiable-answer math with models in the 1.5B-to-7B range, which is a long way from frontier scale and from domains where correctness cannot be checked by a script. The method also requires both the pre-RL and post-RL copies of the teacher, which you have if you trained it and do not if you downloaded it. And it is one entry in a fast-moving line of work -- DOPD and MOPD attacked adjacent problems weeks ago. The idea that the learning signal, not the learned model, is the transferable asset is the part likely to outlive the specific recipe.

Primary source, verified: read the paper → (arXiv 2607.05394)

Ground Truth. (2026, July 15). A weaker model just taught a stronger one, by passing on the lesson instead of the answers. Ground Truth. https://groundtruth.day/news/direct-opd-weak-teacher-stronger-student.html

@misc{groundtruth:direct-opd-weak-teacher-stronger-student,
  title  = {A weaker model just taught a stronger one, by passing on the lesson instead of the answers},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/direct-opd-weak-teacher-stronger-student.html}
}

Topics: distillation · rl-post-training · weak-to-strong · llm-training · research