Learn · Intermediate
Control variates: reducing noise without changing the answer
A control variate is a technique for making a noisy estimate more precise by using a related quantity whose average is known. It matters in machine learning because gradients based on sampled actions or random trajectories can fluctuate wildly, making learning slow even when the average update points in the right direction.
Imagine estimating a bakery’s average daily revenue from a handful of days. You also know the long-run average number of customers. A day with unusually many customers probably has unusually high revenue, even if nothing else changed. Adjusting revenue for the known customer-count deviation can remove some of that day-to-day noise. The adjustment works when the relationship is useful and the reference average is correct.
The basic construction
Suppose X is the random quantity whose mean we want, and Z is another random quantity with a known mean μ. For a fixed coefficient c, define an adjusted quantity:
X_adjusted = X - c × (Z - μ).
Because the average of Z - μ is zero, the adjusted quantity has the same expectation as X. Its variance, however, can be smaller: when X is unusually high and Z tends to be high too, the subtraction cancels part of the fluctuation. A negatively correlated Z can also help if c has the appropriate sign. If Z has no useful relationship to X, the adjustment may do nothing or make the estimate noisier.
For a scalar estimate, the best fixed coefficient in the idealized case is the covariance of X and Z divided by the variance of Z, provided the latter is positive. In practice those quantities must often be estimated. A coefficient fitted too aggressively on a tiny sample may behave badly, and estimating it using the same data can complicate the clean unbiasedness argument. Separate data or careful analysis can help.
Why policy gradients need one
In reinforcement learning, a policy assigns probabilities to actions. An agent samples actions, receives rewards and adjusts the policy to increase expected reward. The sampled return is noisy: two identical decisions can lead to different outcomes because of later choices, opponents or the environment. A single successful episode does not reveal which of its many actions deserved credit.
Ronald Williams’ REINFORCE paper describes a family of stochastic gradient methods that can use reward baselines. In a common form, the update multiplies the gradient of the log probability of an action by the observed return minus a baseline. The baseline predicts what return was ordinary for that situation, so the update emphasizes whether the result was better or worse than expected.
The key identity is that, under the usual regularity conditions, the expected gradient of the log probability of an action drawn from the policy is zero. Therefore, subtracting an action-independent baseline for a given state leaves the expected gradient unchanged. A learned state-value prediction is a common choice. It is an estimate of typical future reward, not a guarantee that every update is correct. The policy-gradient background and Markov decision process lesson supply the surrounding setting.
This distinction is easy to miss: a baseline changes the noise around the expected update, not necessarily the direction of each individual sampled update. A baseline that is too high can make even a positive reward produce a negative advantage; that says the outcome was worse than the baseline, not that the reward itself was negative.
When the guarantee breaks
A baseline that depends on the sampled action in the wrong way generally cannot be subtracted for free. Its contribution may have a nonzero expected gradient and bias the estimator. More elaborate action-dependent control variates are possible, but require a correction. Likewise, if the baseline shares parameters with the policy, an implementation must handle derivatives consistently with the estimator being used; blindly differentiating every part of a surrogate loss can change the intended update.
Control variates do not fix a biased reward, a misleading benchmark or a poor task distribution. They address variance in an estimator under stated assumptions. As John Schulman and colleagues explain in their work on stochastic computation graphs, the computational structure determines which gradient estimators and variance-reduction techniques apply. An estimator can be unbiased yet too noisy to be useful, or low-variance but biased; both properties matter.
How to judge one in practice
Compare the variance of independent gradient estimates with and without the control variate on the same representative distribution. Check learning progress and compute cost too: a sophisticated baseline can consume enough resources to outweigh its savings. Inspect whether the baseline depends on the action and whether its expected contribution is actually zero. Report the assumptions instead of treating “variance reduction” as a synonym for better training.
This idea also helps interpret modern agent training. In the Hugging Face harness-adapter report, task feedback is used to train coding agents in their tool environments. A reward signal is only one part of such a system: how it is sampled, normalized and translated into updates affects stability, while the evaluation still has to establish whether the trained agent transfers to new work. Control variates explain one way to make learning from noisy feedback more efficient; they do not establish that the resulting behavior is useful or reliable.
Williams, Simple statistical gradient-following algorithms for connectionist reinforcement learning
Schulman and colleagues, Gradient Estimation Using Stochastic Computation Graphs
Key questions
What does a control variate do?
Why use a baseline in policy gradients?
Does a control variate guarantee a better estimate?
Cite this
APA
Ground Truth. (2026, November 21). Control variates: reducing noise without changing the answer. Ground Truth. https://groundtruth.day/learn/control-variates-reducing-noise-in-gradient-estimates.html
BibTeX
@misc{groundtruth:control-variates-reducing-noise-in-gradient-estimates,
title = {Control variates: reducing noise without changing the answer},
author = {{Ground Truth}},
year = {2026},
month = {nov},
url = {https://groundtruth.day/learn/control-variates-reducing-noise-in-gradient-estimates.html}
}