News · 2026-08-16

Reinforcement learning changed only 1 to 3 percent of a model's tokens

Reinforcement learning, the expensive final stage that turns a base language model into a reasoning model, alters only 1 to 3 percent of the tokens a model produces, and the token it promotes is always one the base model was already considering. That is the central measurement in a new paper from researchers at the University of Southern California and the US Army Research Lab, who then built a training-free replacement that matches full reinforcement learning at roughly a thousandth of the training cost.

Key facts

• Only 1 to 3 percent of token positions are affected by reinforcement learning, and the promoted token always lies within the base model's own top five alternatives.
• The paper, "Rethinking RL for LLM Reasoning: It's Sparse Policy Selection, Not Capability Learning", was submitted to arXiv on May 7, 2026.
• Authors: Ömer Faruk Akgül, Rajgopal Kannan, Willie Neiswanger, and Viktor Prasanna.
• Primary source: arXiv:2605.06241, with code at github.com/farukakgul/ReasonMaxxer.

The finding lands in the middle of a long-running argument. Labs spend enormous sums on reinforcement learning post-training, and the resulting models are visibly better at multi-step reasoning. The open question has been whether that process teaches the model anything new, or merely reweights what was already in there. This paper's answer is the second, and it comes with numbers.

Here is the setup. Take a base model and a reinforcement-learning-tuned version of it. Let the base model generate a solution step by step, and at every position ask what the tuned model would have chosen instead. Most of the time, the answer is the same word. In the small minority of cases where the two disagree, the tuned model's preference is almost never something exotic. It is usually the base model's second-ranked candidate.

The authors describe the pattern as "a sparse, predictable correction concentrated at high-entropy decision points where the model is uncertain which branch to take." Entropy here is a measure of indecision. When a model is confident, nearly all its probability sits on one word. When it is torn between two ways to continue a proof, the probability spreads out. Those spread-out moments are where reinforcement learning does its work, and almost nowhere else.

Think of it as a hiker who already knows every trail on the mountain. Reinforcement learning does not build new trails. It stands at a handful of forks and points.

The causal test is the part that makes this more than an observation. The researchers applied corrections at only those few high-entropy positions and recovered a large fraction of reinforcement learning's accuracy gain. Applying the same number of corrections at random positions failed. That is the difference between a correlation and a mechanism, and it is why the result is worth taking seriously.

Then comes the practical payoff. If the corrections are sparse, predictable, and locatable using nothing but the base model's own uncertainty, you do not need the reinforcement learning loop to find them. The authors built ReasonMaxxer, which they describe as "a minimal RL-free method that applies contrastive loss only at entropy-gated decision points, using a few hundred base-model rollouts and no online generation." No reward model, no rollout-and-score loop, no cluster.

Across three model families, six model scales, and six math reasoning benchmarks, they report that ReasonMaxxer "matches or exceeds full RL performance while requiring only tens of problems and minutes of single-GPU training, a reduction in training cost of roughly three orders of magnitude."

Their conclusion is stated plainly in the abstract: "These findings reframe reasoning improvement as sparse policy selection, not capability acquisition."

The caveats are real and the authors do not hide them. Everything here is measured on mathematical reasoning, where an answer can be checked mechanically, which is exactly the terrain reinforcement learning with verifiable rewards was built for. Whether the same sparsity holds for open-ended writing, agentic tool use, or safety behavior is untested. The thousand-fold cost figure is also a ratio between estimated training runs rather than an audited bill, and the paper's own tables show it varies enormously depending on which reinforcement learning baseline you compare against. It is a headline, not a constant.

There is also a counterweight in the literature. Other recent work argues that reinforcement learning can move a model's reasoning ceiling if you first find the edge of its competence and aim targeted guidance there, which would make this a result about standard recipes rather than a law about reinforcement learning itself.

Still, the direction is consistent with a run of recent findings, including a model that improved itself by training only where it disagreed with itself. If the useful signal in post-training really is this concentrated, the economics of building a reasoning model look very different from what the current spending implies. Related reading: fine-tuning and LoRA and how AI picks its next word.

Primary source, verified: read the paper → (arXiv 2605.06241)

Ground Truth. (2026, August 16). Reinforcement learning changed only 1 to 3 percent of a model's tokens. Ground Truth. https://groundtruth.day/news/reinforcement-learning-changed-only-1-to-3-percent-of-a-models-tokens.html

@misc{groundtruth:reinforcement-learning-changed-only-1-to-3-percent-of-a-models-tokens,
  title  = {Reinforcement learning changed only 1 to 3 percent of a model's tokens},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/reinforcement-learning-changed-only-1-to-3-percent-of-a-models-tokens.html}
}

Topics: reinforcement-learning · reasoning · post-training · efficiency · research