News · 2026-10-26

A reinforcement-learning critic can supply confidence, under specific conditions

Shujin Wu and Heng Ji report that a value critic already trained during some forms of reinforcement learning can estimate whether a language model will answer correctly. Their method improved answer selection in experiments on two task models, with an average accuracy gain of 5.6 percentage points for one model when choosing among eight responses. The result depends on a specific correctness reward and does not make confidence universally free.

Key facts

• The first arXiv version was submitted October 22, 2026.
• The study trained two eight-billion-parameter task models and tested mathematics, code, tool calling, and general reasoning.
• Selecting the highest-scored answer among eight improved average accuracy by 4.1 and 5.6 percentage points in the two tested models.
• The primary source is Free Lunch for LLMs Confidence, with full methods and results.

The useful idea is an identity between two prediction problems. A confidence estimator tries to predict whether an answer will be correct. A reinforcement-learning critic predicts the future reward expected from a state. If the only reward is one for a correct completed answer and zero for an incorrect one, and the reward is not discounted, those targets coincide.

The authors compress the idea into their subtitle, “The RL Critic Already Knows.” The phrase should be read as shorthand for a learned prediction, not as evidence that a model has privileged access to truth. The critic can only learn from the rewards, examples, and answering policy it actually encounters.

A concrete analogy is a coach estimating whether a player will finish a drill successfully from the current position. If success pays one point and failure pays none, the expected score and the probability of success are the same number. Change the scoring to reward style, speed, or intermediate moves, and the expected score no longer means only the probability of finishing.

That reward condition is the heart of the result. An assistant rewarded for sounding helpful, using fewer resources, or satisfying a complicated grader may have a critic that predicts those combined objectives. It cannot automatically be interpreted as a factual-correctness meter. The interpretation requires matching the value target to the event the user cares about.

The proposed method, Critic-as-Confidence, reads the value estimate at the prompt or a generated prefix. In value-based training, it reuses a component already trained during policy optimization instead of adding confidence-specific training or changing how the answering model behaves. This is different from asking the model to write a percentage in its response.

The experiments use Proximal Policy Optimization, a widely used method that trains an answering policy alongside a value estimate. The broader intuition behind learning useful value signals appears in generalized advantage estimation. Both sources help explain why a training component can contain information useful for a different downstream decision.

The authors evaluated ranking and calibration separately. Ranking asks whether correct answers tend to receive higher scores than incorrect ones. Calibration asks whether a score such as 70% corresponds to success about seven times out of ten. The distinction matters because an ordering can be useful for selecting a response even if its numerical probabilities need adjustment. Our calibration lesson covers that difference.

In the paper's experiments, the critic-based method led all eight tested in-distribution model-and-domain splits on the ranking measure. It also performed strongly on shifted-domain comparisons and calibration. These are author-reported results, not independent evidence that every critic will transfer to every new subject or deployment setting.

The eight-response selection experiment supplies the most concrete operational illustration. Generate several candidate answers, score them with the critic, and choose the highest. The reported gains show that the signal can be useful for selection in the tested setting. But generating eight answers already spends more computation than generating one, so the selection gain should not be confused with a cost-free improvement to a single response.

The title's free-lunch claim has two further limits. First, inference still requires an additional critic computation. Second, some training approaches, including group-relative policy optimization, do not already supply the needed value critic. For those cases, the authors fit a value model on existing rollouts, which adds a training stage.

Distribution shift is another boundary. A new question type, changed decoding strategy, or a policy that has moved beyond the critic's training conditions can weaken the score's meaning. A critic trained against an imperfect grader predicts that grader's reward, including its mistakes. It is not an independent fact-checker.

The paper identifies a useful source of confidence information that a training pipeline may already possess. It does not establish a user-facing confidence switch for existing chatbots or a safety guarantee for medical decisions. The practical implication is to inspect the reward and policy conditions before assigning a probability interpretation to the value estimate, then validate that interpretation against actual outcomes.

Primary source, verified: read the paper → (arXiv 2610.31820)

Ground Truth. (2026, October 26). A reinforcement-learning critic can supply confidence, under specific conditions. Ground Truth. https://groundtruth.day/news/rl-critic-confidence-estimator-conditional-free-lunch.html

@misc{groundtruth:rl-critic-confidence-estimator-conditional-free-lunch,
  title  = {A reinforcement-learning critic can supply confidence, under specific conditions},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/rl-critic-confidence-estimator-conditional-free-lunch.html}
}

Topics: research · reinforcement-learning · confidence · calibration · evaluation