Learn · Intermediate

Direct Preference Optimization: skipping the reward model entirely

Direct Preference Optimization, or DPO, trains a language model to match human preferences using nothing more than a standard supervised loss and a frozen copy of the original model. It throws away the two most troublesome components of the usual recipe: the separate reward model, and the reinforcement learning loop. The 2023 paper that introduced it, by Rafael Rafailov and colleagues at Stanford, carries a title that gives away the whole trick: Your Language Model is Secretly a Reward Model.

The problem it replaced

A pretrained language model predicts plausible text. It does not, on its own, prefer helpful answers to unhelpful ones, because nothing in next-token prediction encodes what a person actually wants. The standard fix is reinforcement learning from human feedback, which OpenAI described in detail in the InstructGPT paper and which built on Paul Christiano's earlier work on learning from human preferences.

That recipe has three stages. Fine-tune on good examples. Collect human comparisons, where a labeler sees two responses and says which is better, and train a separate reward model to predict those judgments. Then use reinforcement learning, usually PPO, to update the language model so it maximizes the reward model's score, with a penalty term to stop it drifting too far from where it started.

Stage three is where the pain lives. You are now training two large neural networks against each other, one of which is a learned and imperfect judge. The optimizer will happily discover that a certain phrasing tickles the reward model without being any good, which is reward hacking in its purest form. Runs are unstable, hyperparameters are finicky, memory footprint is large because you are holding several model copies at once, and reproducing someone else's result is notoriously hard.

The insight

Rafailov's group asked what the optimal policy looks like in closed form. For the standard objective, maximize reward while staying close to a reference model, there is a known answer: the optimal policy is the reference model reweighted by the exponential of the reward. That is a textbook result.

The move is to invert it. If the optimal policy is a function of the reward, then the reward is a function of the policy. Rearrange the algebra and the reward for a response turns out to be, up to a constant, the log-ratio between how likely your model is to produce it and how likely the frozen reference model is. Every language model is therefore already an implicit reward model. You do not need to train one.

Substitute that expression into the standard loss used for preference comparisons, and the reward function cancels out of the problem entirely. What remains is a plain classification loss over pairs: raise the likelihood of the preferred response relative to the reference, lower the likelihood of the rejected one, with the size of the update scaled by how wrong the model currently is about which is better. No reward model. No sampling. No reinforcement learning. Just a batch of pairs and gradient descent.

An analogy: imagine judging a cooking contest. The RLHF approach trains a mechanical judge to imitate the tasters, then has the chef cook thousands of dishes to please the machine. DPO notices that the chef's own relative enthusiasm for each dish already encodes the same information, and simply tells the chef to shift toward the dishes tasters preferred and away from the ones they did not.

What it costs

DPO's simplicity has a real price, and understanding it explains why the biggest labs did not abandon reinforcement learning.

DPO is offline. It learns only from the pairs already in your dataset, all generated before training started. Reinforcement learning is online: at every step it samples fresh responses from the current model and gets them scored, so it discovers and corrects the specific failure modes the model develops during training. DPO cannot see those, because nobody labeled them. This is the on-policy versus off-policy distinction, and it is the central tradeoff.

DPO also has failure modes of its own. It can push down the probability of the rejected response so aggressively that it drags the preferred one down too, since the two are usually similar text. It is sensitive to the beta parameter controlling how far the model may drift from the reference. And it inherits every bias in the preference data with no intermediate reward model to smooth over label noise.

Where it landed

DPO won the middle of the market decisively. It is the default for open-weight fine-tuning because it runs on modest hardware, converges predictably, and needs no distributed reinforcement learning infrastructure. Variants followed quickly, including KTO, which drops the requirement for paired comparisons and learns from single thumbs-up or thumbs-down labels, a much easier thing to collect in production.

At the frontier, the picture is mixed and moving. Preference-based methods like DPO are often used for style, tone and refusal behavior, while reinforcement learning with verifiable rewards and GRPO handle reasoning, where an automatic checker can grade an answer without any human in the loop at all.

The durable lesson from DPO is not really about preferences. It is that a chunk of machine learning complexity can sometimes be dissolved by algebra rather than engineered around. Two stages of infrastructure disappeared because someone rearranged an equation.

Ground Truth. (2026, August 15). Direct Preference Optimization: skipping the reward model entirely. Ground Truth. https://groundtruth.day/learn/direct-preference-optimization.html

@misc{groundtruth:direct-preference-optimization,
  title  = {Direct Preference Optimization: skipping the reward model entirely},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/direct-preference-optimization.html}
}

Topics: rlhf · alignment · fine-tuning · post-training · preference-learning