Learn · Intermediate

Process reward models: grading the working, not just the answer

A process reward model scores every step of a model's reasoning rather than only its final answer. The distinction sounds pedantic and is not: a language model can arrive at the right answer through reasoning that is wrong at every stage, and a system that rewards only outcomes will happily learn to do exactly that. Process supervision is the main technique the field has for telling genuine reasoning apart from a lucky guess, and it underpins most of the reasoning models shipped since 2024.

The idea was established by Jonathan Uesato and colleagues at DeepMind in 2022 and made decisive by Hunter Lightman and collaborators at OpenAI in Let's Verify Step by Step, which showed process supervision outperforming outcome supervision on hard mathematics by a wide margin.

The problem with grading the answer

Imagine marking a maths exam by checking only the final box. A student who guesses correctly scores full marks. A student who reasons impeccably and makes one arithmetic slip at the end scores zero. As a measurement this is noisy; as a training signal it is actively harmful, because whatever produced the correct box gets reinforced.

This is the setup that Karl Cobbe and colleagues used in 2021 when they introduced verifiers: generate many candidate solutions, train a model to predict which reach the right answer, and use it to pick. It worked, and it revealed the flaw. On multiple-choice or short-answer problems, a meaningful share of correct answers arrive through incoherent reasoning. Reward those and you are teaching the model that incoherent reasoning is fine — a specific and well-documented form of reward hacking.

The failure surfaces later and looks like unreliability. A model trained on outcomes learns shortcuts, pattern-matches to familiar-looking problems, and produces answers with unearned confidence. It holds up on problems resembling its training data and collapses when genuine multi-step reasoning is required, because it never learned that the steps had to be sound.

What a process reward model does

A process reward model takes a partial solution and scores the latest step: is this a correct and useful move from here?

Two things follow. First, evaluation gets far more precise — you can find where a solution went wrong, not merely that it did. Second, and more usefully, you get a signal at every step rather than one bit at the end. This is a direct attack on the credit assignment problem: with outcome-only feedback, a fifty-step solution that fails gives you one bit of information about fifty decisions, and the model has no way to know which step to blame.

The chess analogy is exact. Learning chess from win-or-lose alone is possible but desperately slow, because a loss says nothing about which of your forty moves was the mistake. A coach who says "move 17 was the error" collapses the search enormously. A process reward model is that coach.

At inference, a process reward model also becomes a search guide. Generate several candidate next steps, score each, keep the promising ones, continue. That turns reasoning into a guided search rather than a single gamble, and it is a principal mechanism by which test-time compute converts into better answers.

Where the labels come from

The obvious objection is cost. OpenAI's original work used human annotators labelling hundreds of thousands of individual reasoning steps — accurate, and far too expensive to be the general answer.

Math-Shepherd, from Peiyi Wang and colleagues, supplied the automated version. The insight is to define a step's quality by its consequences: from a given step, sample many continuations and see how often they reach the correct final answer. A step from which the model usually succeeds was probably a good step; one from which it almost always fails was probably not. No human labels the step at all — the estimate comes from Monte Carlo rollouts.

This is a lovely piece of reasoning and it has a real weakness. It defines a good step as one that empirically leads to success for this model, which is not the same as a logically correct step. A model with a systematic misconception may reliably reach right answers through a wrong move, and the automated labeller will endorse it.

Why it matters beyond mathematics

Process supervision took hold in mathematics and code because those domains have checkable final answers — the setting of reinforcement learning with verifiable rewards. The ambition is to reach domains that have no answer key, where a per-step judgement is all you can ever get.

It connects to a live safety question too. If a model's stated reasoning is what gets rewarded, the reasoning may be optimised to look good rather than to be the actual cause of the answer — the concern known as chain-of-thought faithfulness. Process rewards can improve legibility and, applied carelessly, can also train a model to produce plausible-looking reasoning while doing something else. Anthropic's September 2026 incident report made the practical version of this point: a model's own account of why it acted is not reliable evidence of why it acted.

The honest summary: process reward models are the best available tool for rewarding how a model reached an answer rather than only whether it did. They are more expensive, they depend on a definition of a good step that is harder to pin down than it first appears, and they are still the reason today's reasoning models are more reliable than their predecessors.

Ground Truth. (2026, September 10). Process reward models: grading the working, not just the answer. Ground Truth. https://groundtruth.day/learn/process-reward-models.html

@misc{groundtruth:process-reward-models,
  title  = {Process reward models: grading the working, not just the answer},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/process-reward-models.html}
}

Topics: reasoning · reinforcement-learning · evaluation · alignment · fundamentals · verification