Learn · Intermediate

Contextual bandits: learning which choice helps when you only see one outcome

A contextual bandit is a learning system that chooses an action using the current situation and then observes the outcome of the action it selected. It balances using choices that already look good with trying alternatives that could be better. The method matters whenever a system must learn from incomplete feedback, such as selecting a recommendation, a model route or a memory for an agent.

Imagine a librarian choosing one reference card for each visitor. The visitor arrives with a question, and the librarian can see features of that question before choosing. Later, the librarian learns whether the visit ended successfully. The missing information is what would have happened if a different card had been offered—or if the visitor needed no card at all.

That is the central difference from ordinary supervised learning. In a classification dataset, a training example may come with the correct category. In a bandit interaction, the learner generally sees a reward for its selected action, not labels for every possible action. This partial-feedback structure makes learning harder even when the individual prediction model is simple.

The context is the information available before the choice. The action is what the learner selects. The reward is the measured outcome. A policy maps contexts to action choices, potentially with randomness. Those definitions separate a genuine contextual decision from a system that simply recommends the same historically popular option to everyone.

Lihong Li, Wei Chu, John Langford and Robert Schapire gave an influential application in their 2010 personalized-news paper. Their work connects recommendation to contextual bandits: an article choice depends on the user and situation, while feedback arrives for what was shown. A familiar user-facing feature therefore becomes a problem of deciding under uncertainty, rather than predicting an answer from fully labeled examples.

The exploration problem follows immediately. Suppose the librarian initially tries card A on easy questions and receives several successes. Card B remains mostly unused. A greedy system could keep choosing A forever, even if B would help more on the difficult questions that now arrive. Success counts alone mix action quality, question difficulty and the history of who received which option.

An optimistic strategy gives uncertain choices some additional priority, within a defined model and its assumptions. Another strategy samples plausible beliefs about action quality and chooses according to the sampled belief. Shipra Agrawal and Navin Goyal analyze the latter idea in their contextual Thompson-sampling paper. Neither strategy makes every individual exploratory choice beneficial; its purpose is to improve what the learner can discover over time.

A common objective is regret: the accumulated reward the learner misses relative to a specified comparison policy. That comparison must be stated carefully. A guarantee against the best policy in a restricted class is not a guarantee against an all-knowing decision maker. Assumptions about the reward model, available actions and how conditions change determine what a theoretical result actually covers.

Contextual bandits also differ from the full reinforcement-learning setting. A basic bandit formulation focuses on the immediate consequence of a choice, without modeling the whole chain of future state changes. That makes it simpler, but potentially incomplete when today's action changes tomorrow's opportunities. Choosing a memory to retain can affect what will be available much later, which is more demanding than choosing a card for an isolated visit.

Today's TokenWorth paper supplies a useful neighboring problem. Its controller estimates which experience to retrieve and which records to keep, using task outcomes under resource constraints. It should not automatically be relabeled as a standard contextual-bandit algorithm. The connection is the feedback limitation: observing success after a retrieval does not establish that the retrieved experience caused the success.

Evaluation needs to respect that limitation. A log from an old policy tells us what happened after its choices, rather than what every alternative would have achieved. Randomized exploration with recorded selection probabilities can support principled estimates of another policy's reward, provided the required coverage and assumptions hold. If an action was never eligible or never selected, the log cannot conjure its missing outcome. Very small selection probabilities can also produce unstable estimates.

This is why the issue overlaps credit assignment without being the same lesson. Credit assignment asks which part of a process deserved credit. Contextual bandits specify a decision-and-feedback structure and develop policies for learning within it. Before applying either idea, define the decision, the outcome window and which alternatives the evidence actually covers. Then ask whether the measured reward represents the useful behavior you wanted, rather than an easy proxy the system can learn to exploit.

Ground Truth. (2026, October 20). Contextual bandits: learning which choice helps when you only see one outcome. Ground Truth. https://groundtruth.day/learn/contextual-bandits-learning-which-choice-helps.html

@misc{groundtruth:contextual-bandits-learning-which-choice-helps,
  title  = {Contextual bandits: learning which choice helps when you only see one outcome},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/contextual-bandits-learning-which-choice-helps.html}
}

Topics: contextual-bandits · reinforcement-learning · exploration · evaluation · agent-memory