Learn · Intermediate

Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization, or GRPO, is the reinforcement learning method behind most of today's reasoning models. Its core idea is simple: instead of training a second neural network to judge how good an answer is, GRPO samples a whole group of answers to the same question and scores each one by how it stacks up against the group's average. It was introduced by DeepSeek in early 2024 and became famous as the algorithm that trained DeepSeek-R1 to reason step by step.

To see why GRPO matters, start with the method it replaced. The standard reinforcement learning algorithm for language models was PPO (Proximal Policy Optimization). PPO needs a 'critic' - a second neural network, usually as large as the model being trained, whose job is to predict a baseline: how good an answer at this point is expected to be. You then reward the model for beating the baseline and penalize it for falling short. The critic works, but it is expensive. It roughly doubles the memory you need during training, and it is itself hard to train well.

GRPO's insight, introduced in the DeepSeekMath paper, is that you don't need a learned critic if you can just ask the model the same question several times. For a given prompt, GRPO samples a group of answers - say eight or sixteen - and scores each with a reward (for math, simply: is the final answer correct?). Then it computes the group's average reward and uses that as the baseline. An answer that scored above the group average gets pushed up; one below gets pushed down. The 'advantage' of each answer is just its reward minus the group mean, usually divided by the group's standard deviation. That is the whole trick: the group is its own critic.

An analogy: imagine grading a classroom of students on an essay with no answer key. PPO is like hiring a professional grader (the critic) and training them for years to predict what a good essay looks like. GRPO is like handing the same prompt to eight students, seeing which essays came out best relative to the others in that batch, and rewarding the standouts. You never need the expensive professional grader - the batch calibrates itself.

This 'relative to the group' design is why GRPO pairs so well with reasoning tasks. On math or code, you can automatically check whether the final answer is right, giving you a clean reward without any human labeling - the approach known as reinforcement learning with verifiable rewards (see RL post-training). GRPO turns that binary signal into a training gradient: within a group where some answers are right and some wrong, the right ones get reinforced. Over many steps, the model learns the reasoning paths that tend to reach correct answers. Because rollouts are generated by the current model, GRPO sits mostly on the on-policy side of the spectrum.

There is a subtle and important detail in that division by the group's standard deviation. A recent unification result, reported on Ground Truth, proved that GRPO, its 'done right' variant Dr. GRPO, and the DAPO method are really three operations on a single number: the group reward standard deviation, which measures how much the sampled answers disagree. GRPO divides by it, Dr. GRPO removes the division, and DAPO discards groups where all answers agree (and so carry no learning signal). That division is not cosmetic - it decides where the model focuses its learning. GRPO's division subtly gives extra weight to the hardest and easiest prompts; removing it (Dr. GRPO) weights all prompts more evenly. It also has a striking practical consequence: at the common group size of eight, a large fraction of prompts - where all sampled answers happen to agree - produce no learning signal at all.

Why it matters: GRPO made high-quality reasoning training dramatically cheaper and simpler, which is a large part of why capable reasoning models proliferated so quickly. By dropping the critic, it cut the hardware cost of RL post-training and removed a fragile moving part. The honest caveat is that GRPO is not magic - it can still suffer the classic failure modes of RL, including reward hacking (exploiting the reward without truly solving the task) and the entropy collapse where a model becomes repetitive and overconfident. And its behavior depends heavily on details practitioners once treated as plumbing: the group size, the normalization, and how the sampled answers are generated. As the field has learned this year, those details are the algorithm, not an afterthought. See also training vs inference and the broader transformer foundations these methods build on.

Ground Truth. (2026, July 6). Group Relative Policy Optimization (GRPO). Ground Truth. https://groundtruth.day/learn/group-relative-policy-optimization.html

@misc{groundtruth:group-relative-policy-optimization,
  title  = {Group Relative Policy Optimization (GRPO)},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/group-relative-policy-optimization.html}
}

Topics: reinforcement-learning · rl-post-training · grpo · reasoning · training