Learn · Intermediate

Self-Play: How a Model Gets Better by Competing With Itself

Self-play is training a model by having it play against copies of itself. The trick it buys you is an opponent that gets stronger at exactly the rate you do, which means the training signal never becomes too easy or too hard, and the supply of training data is limited only by how much compute you are willing to spend. It is how DeepMind's AlphaGo Zero reached superhuman Go from nothing but the rules, and it is the idea that several 2026 training methods are trying to drag out of games and into open-ended work.

The problem it solves

Two things run out when you train a model on human data: the data, and the difficulty.

The data problem is obvious. There are only so many recorded Go games, and the strongest human play is a ceiling, not a floor. The difficulty problem is subtler and more important. If you train an agent against a fixed opponent, it learns to beat that opponent and then stops learning, because nothing about the situation demands more. If you train against an opponent that is far too strong, it loses every game and learns nothing at all, because the reward is never anything but zero.

What a learner needs is an opponent slightly better than itself, continuously, forever. There is exactly one opponent guaranteed to be that: itself.

How it works

The loop is startlingly simple. Take the current model. Have it play a game against a copy of itself. Record the moves and who won. Train on the result -- reinforce the moves that appeared in winning games, discourage the ones that appeared in losing games. Now you have a slightly better model. Make a copy. Play again.

Tennis is the intuition. A player who only rallies against a wall gets very good at the wall's one shot. A player who rallies against a mirror of themselves faces every improvement they make, from the other side of the net, the moment they make it. Every trick they discover is immediately a trick that gets used on them, so the only way to keep winning is to discover the counter.

David Silver and colleagues at DeepMind showed in the AlphaGo Zero paper that this is enough on its own. The system started with random play and the rules of Go, and no human games at all, and surpassed the version that had learned from human experts. AlphaZero then applied the same recipe to chess and shogi without changing the algorithm, which was the stronger claim: self-play was not a Go technique.

Two components make the loop work in practice. The first is a way to search ahead, so that the moves being trained on are better than the model's raw instinct -- the search acts as a teacher the model then distills, which connects to distillation. The second is a pool of past opponents rather than only the newest one, because a model that trains solely against its current self can spiral into a strategy that beats only itself. AlphaStar, described in DeepMind's StarCraft II paper, maintained an entire league of past and specialized agents for exactly this reason.

What emerges

The most striking property of self-play is that behaviour nobody designed shows up anyway. In Emergent Complexity via Multi-Agent Competition, simple simulated bodies given only a competitive objective developed tackling, ducking, and feinting. In OpenAI Five, agents trained on Dota 2 developed team coordination and long-horizon strategy that the reward function never mentioned. Competition is a curriculum generator. Any weakness one copy has becomes an exploit the other copy finds, and the exploit becomes the next thing to fix.

The corresponding failure is that competition also finds exploits in your simulator. If the physics engine lets an agent phase through a wall, self-play will discover that long before it discovers a good strategy. This is reward hacking with an extremely diligent attacker.

Getting it out of games

Games have one property that makes self-play possible: the environment knows who won. No human, no judge model, no rubric. That is the same property behind reinforcement learning with verifiable rewards, which is why maths and code post-training works so well and summarization does not.

Two directions are trying to widen it. The first keeps the game but changes the players: Zixiang Chen and colleagues' Self-Play Fine-Tuning sets a language model against its own earlier outputs, training it to distinguish its previous generations from human-written text, so the model improves by out-running its own past self rather than by beating another player.

The second manufactures a checkable fact where none existed. A 2026 method called SpyRL runs the party game Who Is the Spy: several copies of a model perform the same task, one of them secretly working from a document with an important passage removed, and a detector votes on which one was handicapped. The environment recorded the answer, so the reward is exact -- and if the handicapped copy's output is detectably worse, then playing the game well requires producing good work. That is the whole bet, and it is a real one: the score is exact, but whether the score tracks quality is empirical. Our reporting on what happened when that method's code was read closely is also a lesson in checking the implementation, not just the idea.

The takeaway

Self-play converts compute into training data and difficulty into a curriculum, without a human in the loop. It requires an environment that can score an outcome by itself, which is why it conquered board games first. Everything happening now at the frontier of RL post-training is an attempt to invent that scorer for domains that never had one.

Ground Truth. (2026, August 3). Self-Play: How a Model Gets Better by Competing With Itself. Ground Truth. https://groundtruth.day/learn/self-play.html

@misc{groundtruth:self-play,
  title  = {Self-Play: How a Model Gets Better by Competing With Itself},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/self-play.html}
}

Topics: self-play · reinforcement-learning · training · fundamentals · multi-agent