Learn · Intermediate

Importance sampling: evaluating a new AI policy from an old policy’s logs

Importance sampling estimates an average under one probability distribution using observations collected under another. In AI policy evaluation, it reweights old actions by how likely a proposed policy would have been to take them. It can estimate a new policy’s expected reward without deploying it, provided the old logs cover its choices and record the necessary probabilities.

Suppose a system chooses which model should answer each support request. An old router sends 80 percent of eligible requests to model A and 20 percent to model B, choosing randomly in this simplified example. A proposed router would split them evenly. The old log contains too many A outcomes and too few B outcomes to average directly for the new router.

For each recorded choice, compute the target probability divided by the logging probability. An A outcome receives weight 0.5 divided by 0.8, or 0.625. A B outcome receives weight 0.5 divided by 0.2, or 2.5. Average the weighted rewards over logged requests. Under the stated randomization and a stable outcome process, the expectation matches the target policy’s value.

The analogy is correcting a survey that deliberately sampled one neighborhood more often than another. You do not pretend every respondent represents the same share of the population. You account for the sampling design. Here the imbalance comes from the old policy’s action choices, and the target population is the set of choices the proposed policy would make.

In realistic routing, probabilities depend on the request. The logging policy might choose an expensive model more often for difficult questions. Its recorded probability must therefore be conditional on the relevant context, and the target probability must be evaluated for that same context and action. Using overall action frequencies instead can confuse difficulty with model quality.

This extends the partial-feedback problem in contextual bandits. We observe the reward of the action actually taken, not the reward of every alternative. On-policy and off-policy learning concern where training experience comes from. Off-policy evaluation asks a different question: what reward would a specified policy obtain, before deciding whether to train or deploy it?

Coverage is the first hard limit. If the old system never sent a certain class of requests to model B, a proposed policy sending those requests to B has no corresponding observed outcomes there. Dividing by zero does not manufacture evidence. A reward-prediction model may extrapolate, but its answer then depends on modeling assumptions rather than a pure importance-sampling correction.

Variance is the next limit. An action with logging probability one in a thousand can receive an enormous weight when the target often chooses it. A single unusually good or bad outcome can dominate the answer. An estimator can be unbiased across infinitely many repeated logs while remaining too unstable to support a decision from the particular log in hand.

Clipping weights reduces that instability by capping the influence of rare observations, at the cost of introducing bias. Normalizing weights by their observed sum is another common trade-off; it generally changes finite-sample bias while often improving stability. Neither repairs missing support. Effective sample size, often summarized using the squared sum of weights divided by the sum of squared weights, helps reveal when thousands of records carry far less independent information than their count suggests.

Miroslav Dudík, John Langford and Lihong Li’s Doubly Robust Policy Evaluation and Learning combines a reward model with a propensity-weighted correction. The model predicts outcomes, while the correction uses observed residual errors. Under the method’s assumptions, consistency can survive certain errors in either the outcome model or the action-probability model when the other is correct. This does not mean both can be arbitrarily wrong, or that uncertainty disappears.

Long agent workflows make the problem harder. A target may change an entire sequence of decisions and thereby change which future states occur. Trajectory corrections can multiply action-probability ratios across steps, producing extreme variance. A one-step router estimate should not be casually extended to a computer agent operating for hours. WildAWM’s training-world study instead evaluates agents through actual rollouts in resettable environments, illustrating a different way to obtain evidence when execution is feasible.

The practical preparation starts before logging: record contexts, choices, outcomes and the actual selection probabilities, including exploration and filtering. Check coverage, inspect weight concentration and report uncertainty before trusting an apparent improvement. Compare with a fresh controlled test when possible. Importance sampling is a principled bridge from old experience to a new policy’s value, but its strength comes from respecting the data-collection process rather than treating every historical success as interchangeable evidence.

Ground Truth. (2026, October 22). Importance sampling: evaluating a new AI policy from an old policy’s logs. Ground Truth. https://groundtruth.day/learn/importance-sampling-for-off-policy-evaluation.html

@misc{groundtruth:importance-sampling-for-off-policy-evaluation,
  title  = {Importance sampling: evaluating a new AI policy from an old policy’s logs},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/importance-sampling-for-off-policy-evaluation.html}
}

Topics: importance-sampling · off-policy-evaluation · reinforcement-learning · statistics · agents