News · 2026-07-06

The RL 'mirage': the policy you optimize isn't the one you ship

A top-ranked new paper argues that a hidden flaw sits at the heart of how large language models are trained with reinforcement learning: the model you optimize is not the model you deploy, and improving one does not guarantee improving the other. The authors call it a 'mirage,' and propose a fix that was the only method to stay stable in their hardest tests.

Key facts

• The core result: an update that improves the training policy does not imply an improvement to the inference policy you actually serve.
• The proposed method, MIPU, was the only stable approach in high-mismatch tests, averaging 66.7% on math benchmarks while a standard baseline collapsed.
• Titled 'The Mirage of Optimizing Training Policies,' from Tianjin University and Alibaba (arXiv 2606.29526), it was the #1 paper on Hugging Face's daily list.
• Tested on Qwen3-4B and Qwen3-1.7B under FP8-quantized rollout, the setting that most amplifies the mismatch.

Here is the background a non-expert needs. When you train a modern LLM with reinforcement learning, two separate pieces of software are involved. One engine (like vLLM or SGLang) generates the model's attempted answers quickly - this is the 'rollout' or inference side. A different engine (like FSDP or Megatron) computes the gradients that update the model - the training side. In principle they use the same weights. In practice, differences in numerical precision and decoding mean the two engines assign slightly different probabilities to the exact same sentence. The result is a persistent gap: the training policy and the inference policy are subtly different models.

That gap is a form of off-policy mismatch, and prior work tried to patch it on the training side - correcting the sampling ratios, filtering unstable examples, decaying the learning rate. The paper's central insight is that all of those miss the point. Written formally, improving the training policy does not imply improving the inference policy - and the inference policy is the one you deploy. You watch your training metrics climb, feel good, and ship something that may not have improved at all. That is the mirage: you are optimizing the wrong thing.

An analogy: imagine tuning a car on a dynamometer in the lab while it will actually be driven on the road, and the two surfaces disagree just enough that a setting that looks faster on the dyno is slower on the road. Chasing the dyno number is the mirage; the road time is what matters.

The fix, called MIPU, makes the deployed model's improvement the explicit target instead of the training model's. It works in two steps. First, it builds each candidate update referenced to the sampler that actually generated the data, rather than to the training policy - a subtle but important shift that keeps the update honest about where the samples came from. Second, and more novel, it validates each candidate before accepting it: after syncing the update to the inference engine, it estimates whether the deployed policy actually got worse, and if so, it rolls the update back to the previous checkpoint. Propose, then verify.

The experiments used FP8-quantized rollout, because quantized inference makes the two engines disagree the most - a stress test. On Qwen3-4B across five math benchmarks, a standard GRPO baseline reached competitive scores but then degraded and collapsed during continued training; MIPU was the only method that stayed stable, at a 66.7% average. An ablation showed the two steps are complementary: step one alone makes better candidates but accepts everything and drifts; step two alone prevents collapse but holds onto a stale policy; together they improve and stay stable.

The most convincing detail is a control the authors ran to rule out a boring explanation. If step two just helps by rejecting more updates, then rejecting updates at random should help too. It doesn't - random rollback rejected even more updates (67% versus 53.5%) and still collapsed. What matters is not rejecting more, but rejecting the right ones, using the signal about whether the deployed policy actually regressed. This connects to a broader theme in RL post-training research right now: the machinery practitioners treat as plumbing is often the thing that decides whether training works.

The honest caveat: the paper is limited to moderate-scale models (1.7B and 4B parameters) because of compute constraints, and it does not prove a formal guarantee of monotonic improvement - it reduces the risk of accumulating fake gains rather than eliminating it. Whether inference-policy-first training holds up at frontier scale is the open question. But the diagnosis - that the training/inference split quietly corrupts the objective - is the kind of finding that changes how careful teams read their own training curves.

Primary source, verified: read the paper → (arXiv 2606.29526)

Ground Truth. (2026, July 6). The RL 'mirage': the policy you optimize isn't the one you ship. Ground Truth. https://groundtruth.day/news/the-training-policy-you-optimize-isnt-the-one-you-deploy.html

@misc{groundtruth:the-training-policy-you-optimize-isnt-the-one-you-deploy,
  title  = {The RL 'mirage': the policy you optimize isn't the one you ship},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/the-training-policy-you-optimize-isnt-the-one-you-deploy.html}
}

Topics: reinforcement-learning · rl-post-training · training · reasoning · llm