Learn · Intermediate

Bradley–Terry preference models: turning head-to-head choices into scores

The Bradley–Terry model turns a collection of head-to-head choices into scores for the options being compared. It matters in AI because the same mathematical idea can summarize human preferences between model answers and help train a reward model, while making assumptions that must be checked rather than treated as facts.

Suppose reviewers compare two answers to the same question and pick the one they prefer. An answer that wins often against strong alternatives should generally receive a higher score than one that loses often. But a raw win count can be misleading: one answer might have faced easy opponents while another faced difficult ones. The model tries to put the alternatives on a common scale.

The basic mechanism

Assign each option a real-valued score. For two options A and B, the model calculates the probability that A wins as exp(score A) divided by exp(score A) plus exp(score B). The larger the difference between their scores, the higher A's predicted chance of winning. Equal scores imply a 50–50 prediction. The absolute zero is arbitrary: adding the same constant to every score leaves every predicted probability unchanged.

A useful analogy is a sports league in which teams do not all play identical schedules. Wins against high-rated teams convey more information than wins against weaker teams. A statistical fitting procedure finds scores whose predicted match outcomes make the observed choices plausible. The result is a compact summary of the comparisons, not a measurement of some independently existing unit of “quality.”

In language-model training, the options may be two answers to a prompt rather than two fixed teams. A reward model learns a scoring function of both prompt and answer, and training encourages the preferred answer to receive a higher score. Paul Christiano and colleagues’ work on learning from human preferences is a foundational application of pairwise preference modeling to reinforcement learning. OpenAI’s InstructGPT paper describes a later language-model pipeline using ranked outputs and a learned reward model. These are primary examples of preference-based modeling, not evidence that every leaderboard uses the same exact fitting procedure.

From preferences to a model

A typical collection process starts with a prompt and several candidate responses. A reviewer sees a pair and picks the more helpful or otherwise preferred answer according to instructions. The score model receives the prompt and each answer; training adjusts it to assign the chosen response a higher score than its rival. The preference data can be split into training and held-out sets to see whether the learned score predicts choices it did not train on.

Once learned, such a reward model can supply a signal for further optimization. That connection is explained in reward-based fine-tuning. It is important to keep the stages separate: Bradley–Terry-style preference fitting predicts relative choices, whereas reinforcement learning changes the model's behavior using a reward signal. One does not automatically imply the other.

The paired comparisons lesson explains why testing systems on the same tasks reduces some noise. Bradley–Terry adds a way to fit many comparisons to shared scores; pairing does not require this model.

What the scores assume

The model assumes a particular relationship between score differences and win probabilities. It also assumes that a single score is an adequate summary of each option in the specified population of comparisons. Real preferences are often more complicated. One assistant might excel at concise summaries, another at careful mathematical proofs, and reviewers might disagree because they value different things. A single ranking averages over that mixture and can hide reversals across tasks or groups.

Presentation can affect choices too. Reviewers may favor the first answer, longer answers, a confident tone or a familiar style. The original choice might also be a tie or a judgment that both answers are unsafe; forcing a winner discards that information. Randomizing position, allowing ties where appropriate, recording the rubric and testing for these biases help reveal what the scores actually capture.

The comparison schedule matters. If two groups never face each other, their relative positions cannot be identified from those results alone. Even a connected network can be uncertain when few matches link its clusters. Report the comparisons and uncertainty.

Why a high preference score can fail in practice

A preference model learns patterns in the labels it receives. If reviewers regularly reward confident but unverified answers, the model can learn to reward confidence. If a developer optimizes aggressively against that learned score, outputs may exploit its blind spots rather than improve the underlying task. This is a form of reward hacking. A held-out preference test helps measure prediction of similar judgments, but it does not prove factual accuracy, safety or usefulness in every setting.

The same distinction appears in today's finance-agent reporting: Handshake’s simulated finance example describes an apparently complete workbook carrying a faulty tax assumption. A reviewer may prefer a polished answer while missing a consequential mistake. An expert audit or an independently computed check answers a different question from whether an answer was preferred.

Use pairwise scores when comparisons are meaningful, the pool is well connected and the scoring assumptions fit the use. Inspect task-level results, annotator disagreement and uncertainty alongside the aggregate. A rating is a useful map of observed choices; it is not a substitute for the terrain those choices were meant to represent.

Ground Truth. (2026, November 11). Bradley–Terry preference models: turning head-to-head choices into scores. Ground Truth. https://groundtruth.day/learn/bradley-terry-preference-models.html

@misc{groundtruth:bradley-terry-preference-models,
  title  = {Bradley–Terry preference models: turning head-to-head choices into scores},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/bradley-terry-preference-models.html}
}

Topics: preferences · evaluation · reward-models · statistics