Learn · Intermediate

Temporal-difference learning: learning from a better prediction before the outcome arrives

Temporal-difference learning trains a prediction of future reward by comparing it with the next reward plus a new prediction one step later. The agent can therefore learn before the final outcome arrives. It is a foundation of reinforcement learning because many useful decisions receive feedback only after a long sequence of actions.

Imagine a delivery driver estimating the minutes remaining on a route. At one junction the estimate is 20 minutes. After driving for three minutes, the estimate at the next junction is 12. The earlier prediction implied 20 minutes, while the new evidence implies 15: three already spent plus 12 remaining. The driver can improve the earlier estimate without waiting to reach the destination.

That is the central idea Richard Sutton developed in temporal-difference learning. Sutton and Andrew Barto's textbook provides the canonical treatment. The method learns from a discrepancy between successive predictions, anchored by what actually happened between them. It is not simply asking a model to trust its own first guess.

The one-step update

A value function V(s) estimates the expected future return from state s while following a specified policy. A Markov decision process supplies the framework: states, actions, rewards, and transitions. A policy specifies how the agent chooses actions.

After one transition, the temporal-difference error is:

delta = reward + gamma * V(next_state) - V(current_state)

The update moves the current prediction partway toward the new target:

V(current_state) = V(current_state) + alpha * delta

Here gamma controls how much future rewards count, and alpha is the learning rate. At a genuine terminal state, there is no remaining future value. That terminal boundary lets observed outcomes anchor predictions farther back along the sequence.

For a simple numerical example, suppose the current value is 0.4, the immediate reward is zero, the next value is 0.7, and gamma is one. The error is 0.3. With a learning rate of 0.1, the current value becomes 0.43. These are illustrative numbers, not a claim about a measured system.

Why bootstrapping helps

Waiting until the end gives a different method: Monte Carlo learning. It uses the actual accumulated reward from a completed sequence as the target. That avoids substituting a learned estimate for the unseen future, but individual outcomes can be noisy, and a long episode delays the update.

Temporal-difference learning bootstraps from the next prediction. In the simplest tabular policy-evaluation setting, that can use experience efficiently and learn online. However, a poor next-state estimate contaminates the target. The tradeoff is between noisy completed outcomes and potentially biased intermediate predictions.

Multi-step methods lie between the extremes. They collect several actual rewards before bootstrapping from a later value estimate. Trace-based methods combine information across different horizons. John Schulman and colleagues' generalized advantage estimation uses related machinery to construct useful policy-training signals. That connection does not make every reinforcement-learning critic a pure one-step learner.

A value is defined by its reward

A critic is a learned value estimator. It does not inherently measure truth, safety, or permission. If the reward encourages speed, the critic predicts a speed-related return. If the reward combines correctness and style, its value mixes those objectives.

One special case connects directly to today's Critic-as-Confidence study. With no discounting and a single final reward of one for a correct answer and zero otherwise, expected return equals the probability of correctness. Change the reward or the policy conditions, and that interpretation can fail. A mathematical identity about the target also does not guarantee that a learned approximation is accurate or calibrated.

Prediction is different from control

Learning the value of a fixed policy is prediction. Improving which actions the agent selects is control. Q-learning learns values for state-action pairs and bootstraps toward a target involving the best estimated next action. Volodymyr Mnih and colleagues' Atari work combined neural value approximation with replay and a separate target network, making that distinction concrete at scale.

The caveat is stability. Neural approximation, bootstrapping, and learning from a different behavior policy can interact badly. The classic convergence guarantees for simple tabular cases do not automatically carry over to arbitrary neural systems. Replay and target networks help manage training; they are not a universal proof of convergence.

Temporal-difference learning answers a specific engineering question: how can an agent improve a long-term prediction before the long term has happened? It uses the next observation to revise the earlier estimate. Understanding that mechanism makes a critic's usefulness and its limits easier to inspect: ask what return it predicts, how its targets are built, and whether those predictions remain reliable under the policy and data where they will be used.

Ground Truth. (2026, October 26). Temporal-difference learning: learning from a better prediction before the outcome arrives. Ground Truth. https://groundtruth.day/learn/temporal-difference-learning.html

@misc{groundtruth:temporal-difference-learning,
  title  = {Temporal-difference learning: learning from a better prediction before the outcome arrives},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/temporal-difference-learning.html}
}

Topics: fundamentals · reinforcement-learning · value-functions · temporal-difference · training