Learn · Intermediate

Policy entropy: when reinforcement learning makes an AI less willing to try another good path

Policy entropy measures how spread out an AI system’s probabilities are across possible actions. It matters because reinforcement learning can make a model more likely to give its best-known answer while silently making it less able to begin other valid approaches.

Imagine a hiker choosing among several paths down a mountain. A low-entropy policy puts nearly all its belief on one trail. That can be excellent if the trail is certainly safe, but dangerous if a fallen tree blocks it. A high-entropy policy keeps several routes alive. In machine learning, “entropy” is the mathematical summary of that spread: it is high when probability is distributed and low when one option dominates.

A language model has a policy too. At each token, it assigns probabilities to possible next words. In a tool-using agent, the policy also spans actions: search, call a calculator, edit a file, ask a question or stop. Reinforcement learning changes those probabilities based on reward. Policy-gradient methods work by increasing the likelihood of actions associated with good outcomes and decreasing the likelihood of actions associated with poor outcomes. Nothing in that recipe automatically preserves alternative ways to succeed.

That tradeoff is most visible in reward-based fine-tuning. In RLHF and RLVR, a system may get a reward when the final answer verifies. The simplest way to improve the average reward is often to strengthen the already-common successful trajectory. It does not need to preserve every other route that would also work. This is not necessarily a defect. If a factory robot has one safe way to place a part, concentration is useful. The risk arrives when the deployment environment changes or when an evaluator wants a system that can search, recover and generalise.

The recent paper Locked at the Entrance, Open Inside gives a concrete reasoning-model example. Its authors report that reinforcement learning with verifiable rewards improved pass@1 while narrowing the solution space. The largest probability shifts—11 to 16 times larger—appeared before the first arithmetic operation. In other words, the policy did not merely become more confident later in a derivation; it pruned options at the entrance to the problem.

The authors tested whether those paths were gone or merely hard to start. Providing an unselected early prefix restored completion rates in low-access families by more than an order of magnitude. That is a useful distinction. The model may still be capable of executing an alternative solution once placed on the right track, but its learned policy rarely initiates it. An analogy is a library with many books still on the shelves but a recommendation system that sends every visitor to the same aisle.

Why should an engineer care? First, low entropy can make a model brittle under distribution shift. A training set may reward a canonical solution, while a new task needs a noncanonical one. Second, it can hide diversity loss behind a better headline score. A pass@1 gain says the most likely single sample improved; it says little about the set of methods a model still has available. Third, it changes how to debug a failure. If a prompt intervention does not revive the route, the issue might live in early token probabilities or weights rather than the final instructions.

The goal is not to maximise entropy forever. Too much entropy means random, wasteful behavior. Practical approaches include measuring diversity alongside accuracy, sampling multiple solutions where cost permits, rewarding distinct valid strategies, keeping checkpoints before narrow post-training stages, and using targeted interventions. The paper reports that late-layer interpolation with an early checkpoint increased solution coverage by 37% without lowering pass@1 in its setting. That is promising evidence, not a universal fix.

The caveat is scope. The new result is strongest on math reasoning benchmarks and specific 7B and 14B models. It does not prove that every RL-trained model has the same pathology. Still, it teaches a durable evaluation lesson: when a model becomes more accurate, ask whether it also became less willing to look anywhere else.

Ground Truth. (2026, September 7). Policy entropy: when reinforcement learning makes an AI less willing to try another good path. Ground Truth. https://groundtruth.day/learn/policy-entropy-and-mode-collapse-in-reinforcement-learning.html

@misc{groundtruth:policy-entropy-and-mode-collapse-in-reinforcement-learning,
  title  = {Policy entropy: when reinforcement learning makes an AI less willing to try another good path},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/policy-entropy-and-mode-collapse-in-reinforcement-learning.html}
}

Topics: reinforcement-learning · rlvr · diversity · evaluation