News · 2026-11-13

'Leaderboard Hacking' paper: votes, prompts and decoy models can lift an arena ranking by up to 45 places

A preprint posted November 10, 2026 argues that arena-style AI leaderboards can be manipulated, and shows how. Fleisig, Baumann, Hadfield-Menell and Hovy report that three attacks on rankings built with the Bradley-Terry model can each move a model 15 to 20 places and, combined, as much as 45 places. The paper's title says it plainly: "Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation."

Key facts

• Who and when: Fleisig, Baumann, Hadfield-Menell and Hovy, arXiv 2611.14769, submitted November 10, 2026 (cs.AI). No accepted venue is listed.
• Headline: each attack moves a model 15 to 20 ranks; combined up to 45; the attacks boost nearly every tested model.
• Scale: the experiments cover 55 models, and computing the best nomination for all 55 takes under five minutes on one CPU.
• Status: no replication and no response from any arena operator had been checked.

How arena leaderboards work

Platforms such as arena-style chatbots show a visitor two anonymous answers, ask which is better, and convert the pile of pairwise votes into a ranking using the Bradley-Terry model, the statistical model our lesson on Bradley-Terry preference models explains. The appeal is that real people choose the prompts and the winners. The paper's point is that this openness is also an attack surface. It says the weakness applies to any preference-based evaluation using Bradley-Terry that allows open prompting, voting or model submission, a theme our earlier piece the leaderboard is lying approached from a different angle.

The three attacks

Strategic voting. Individual votes boost a target model's rank, including votes cast on matches that do not involve the target at all. Because a ranking is relative, pushing down a model's rivals is as good as lifting the model.

Strategic prompting. The attacker chooses prompts on which the target model is likely to win. This needs the attacker to know which model produced which answer, and the paper shows methods for de-anonymizing responses, using log-likelihood for open-weight models and character-level similarity for closed API models. Those methods come from an earlier read of the paper and were not re-confirmed.

Strategic nomination. The attacker nominates irrelevant models to enter the arena so the target gains rank. The best nomination across all 55 models takes under five minutes of one CPU's time, so this is cheap.

Combined, the paper reports gains of up to 45 places. Each of the three alone moves a model 15 to 20.

A telling detail

The full text notes that the strongest Claude model appears in more battles than any other model, about 14,000 against about 10,000 for the runner-up. That is a measurement of exposure, not manipulation, and the paper does not claim more than that.

The defense, and its limits

The authors try a harmonic Borda weighting (named Dowdall, after a variant used in Nauru's elections). It lowers the largest possible boost but slightly raises the mean possible boost from 2.74 to 2.81 places. In other words, it narrows the worst case while making the average attack slightly better, so it is a trade-off, not a cure. Other defenses and the authors' own caveats were not re-checked against the text. Our lesson on voting rules and why ranking by votes can be gamed explains why no voting rule is fully manipulation-proof.

Why this is a security story

This is a manipulation attack on a measurement system that buyers, investors and journalists rely on. A model maker that wants a higher spot does not need to improve the model, only to steer the evaluation. The same logic applies to any benchmark that accepts open inputs. For readers choosing a model from a ranking, the practical lesson is to treat a single arena rank as one signal, check what the ranking measures, and prefer measurements that hold the prompts fixed.

Caveats

This is a preprint, not peer-reviewed, and the dossier-level checks covered the abstract and selected numbers in the full text. The paper's own claims about how little voting it takes, how effects scale with dataset size, and the other defenses were not re-verified here. No arena operator response was found. The result is that preference leaderboards deserve adversarial testing the way any security-relevant system does.

Primary source, verified: read the paper → (arXiv 2611.14769)

Ground Truth. (2026, November 13). 'Leaderboard Hacking' paper: votes, prompts and decoy models can lift an arena ranking by up to 45 places. Ground Truth. https://groundtruth.day/news/leaderboard-hacking-paper-says-votes-prompts-and-decoy-models-can-lift-a-rank-up-to-45-places.html

@misc{groundtruth:leaderboard-hacking-paper-says-votes-prompts-and-decoy-models-can-lift-a-rank-up-to-45-places,
  title  = {'Leaderboard Hacking' paper: votes, prompts and decoy models can lift an arena ranking by up to 45 places},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/news/leaderboard-hacking-paper-says-votes-prompts-and-decoy-models-can-lift-a-rank-up-to-45-places.html}
}

Topics: cybersecurity · ai-security · evaluation · leaderboards · adversarial · bradley-terry · manipulation