Learn · Intermediate

Credit assignment: figuring out which step deserved the reward

Credit assignment is the problem of deciding which of an agent's decisions actually caused the outcome. An agent runs fifty commands in a terminal and at the end a test suite either passes or fails. That is one bit of feedback for fifty choices. Which command was the good one? Which was the mistake that doomed everything after it? Until you can answer that, you cannot learn much from the episode - and this, more than model size or data volume, is what makes training long-horizon agents hard.

Richard Sutton and Andrew Barto named the problem in its modern form, and the standard framing is the Markov decision process: a sequence of states, actions, and rewards. The difficulty is that the rewards are usually sparse and delayed. In chess you find out you lost after forty moves, and the losing move was probably move twelve. In supervised learning, by contrast, every single input carries its own label, which is why backpropagation works so cleanly there - the error is attributable to the exact prediction that produced it.

The oldest and still most important tool is the value function: a learned estimate of how good a situation is, independent of what happens next. Once you have one, you stop asking "did this episode end well?" and start asking a much sharper question - "did this action leave me better off than expected?" That difference is called the advantage, and it is the workhorse of modern policy-gradient methods. John Schulman and colleagues' generalized advantage estimation gives you a dial between two extremes: attribute the outcome entirely to the final reward, which is unbiased but extremely noisy, or lean on your value estimates at every step, which is stable but only as good as those estimates. Almost every practical system, including PPO and the algorithms behind RL post-training of language models, sits somewhere on that dial.

The intuition is a sales pipeline. A deal closes in March; who gets the commission? Pay only the person who signed it and you starve every earlier step. Split it evenly across everyone who touched the account and you reward the people who did nothing. A value function is the analyst who can say what each account was worth when it landed on each desk - and the commission goes to whoever raised that number.

There are other angles of attack, and the ones that matter are the ones that manufacture signal where none existed.

Relabel the goal. Hindsight Experience Replay makes the observation that a failed attempt is a successful attempt at a different goal. The robot missed the target and knocked the block somewhere else? Pretend that spot was the target. Now the trajectory has a positive reward and something to learn from. Cheap, and startlingly effective when rewards are near-nonexistent.

Decompose the return. RUDDER trains a model to predict the final return as the episode unfolds, then treats the moments where the prediction jumps as the moments that mattered, redistributing reward there. Turn one delayed payment into many immediate ones.

Score every step directly. In language models this became the process reward model. Hunter Lightman and colleagues at OpenAI showed that supervising each step of a chain of reasoning outperforms supervising only the final answer, because outcome supervision rewards a lucky guess arrived at through nonsense. Grading the work, not just the answer, is credit assignment by brute force - and it is expensive, because someone or something has to grade every line.

Which is exactly where 2026's agent research is concentrating. Recent work assigns credit by recursively updating a belief about which turns helped, or by working backward from a known answer to recover the clues a good search should have found, so that even a failed trajectory contributes usable signal on the steps that were right. The shared instinct is that throwing away everything about a failed episode is enormous waste, because most failed episodes contain many correct steps and one bad one.

Two distinctions to keep straight. First, credit assignment is not reward hacking. Credit assignment is distributing a correct reward across the steps that earned it; reward hacking is when the reward itself measures the wrong thing. Bad credit assignment makes learning slow. A bad reward makes learning confidently wrong. Second, denser credit is not automatically better: a per-step signal that is subtly miscalibrated teaches subtly wrong behavior at every step, and if a model is doing the grading, its systematic biases become the agent's systematic biases. This is why the reliability of model judges has become an urgent question rather than an academic one.

The short version: the reward tells you whether. Credit assignment tells you which. Everything difficult about training an agent lives in the gap between those two.

Ground Truth. (2026, August 7). Credit assignment: figuring out which step deserved the reward. Ground Truth. https://groundtruth.day/learn/credit-assignment.html

@misc{groundtruth:credit-assignment,
  title  = {Credit assignment: figuring out which step deserved the reward},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/credit-assignment.html}
}

Topics: fundamentals · reinforcement-learning · agents · training · reward-models