Learn · Beginner
Borda count and manipulable rankings: why no vote-based leaderboard is gaming-proof
A voting rule turns many people's preferences into one ranking, and every sensible rule can be manipulated in some circumstances. That is why open, vote-based AI leaderboards have to be designed with attackers in mind, and why a single arena rank is best read as one signal among several. A November 2026 preprint by Fleisig, Baumann, Hadfield-Menell and Hovy made the point concrete for AI model rankings.
Ranking by votes, three ways
Imagine five voters ranking three models, A, B and C. The simplest rule, plurality, counts each voter's first choice. It ignores everything below the top, so a model most people rank second can lose to one that a few people love and the rest rank last.
The Borda count, named for the eighteenth-century French mathematician Jean-Charles de Borda, uses the whole list. With three options, first place earns 2 points, second earns 1 and last earns 0, and the totals decide. A broadly liked model beats a polarizing one. Dowdall's variant, used in the Pacific island nation of Nauru's elections according to the leaderboard-hacking paper, gives points of 1, 1/2, 1/3 and so on, which favors top positions more strongly.
A third approach, used by arena-style AI leaderboards, collects pairwise comparisons, "answer 1 is better than answer 2," and fits a statistical model to estimate each model's strength. The Bradley-Terry model is the standard choice, explained in our lesson on Bradley-Terry preference models. The Chatbot Arena paper describes this design for language models.
Why none of them is safe
Two classic results explain why. Arrow's impossibility theorem says no ranking rule can satisfy a short list of fairness conditions all at once. The Gibbard-Satterthwaite theorem is closer to security: any deterministic rule that picks among three or more options, and is not a dictatorship, is manipulable, meaning some voter can do better by lying about their preferences. These theorems do not say every election is easy to hack, only that no rule removes the possibility.
For an arena, the attack surface is wider than a ballot box, because participants may also choose the questions and add candidates. The leaderboard-hacking paper describes three attacks: strategic voting, where votes lift a target model even when cast on matches that do not involve it; strategic prompting, where the attacker picks prompts the target is likely to win; and strategic nomination, where irrelevant models are added to the arena to raise the target's rank. Each can move a model 15 to 20 places, and combined as much as 45, across the 55 models studied.
Defenses are trade-offs
The authors test a Dowdall-style harmonic weighting. It lowers the largest possible boost but slightly raises the average possible boost, from 2.74 places to 2.81. That is typical of the field: a defense reduces one risk and shifts another. Practical mitigations include fixing a hidden or rotating set of prompts, limiting how many votes any one account can cast, hiding which model wrote which answer, and watching for statistical oddities. Anonymity itself can be defeated, since the paper shows methods for identifying which model wrote an answer.
How to read a leaderboard
- Ask what the ranking measures and who can influence it.
- Prefer rankings with fixed, private test sets for decisions that matter, and use arenas as a signal about human preference.
- Compare across several sources; if a model ranks high on one list only, be curious why.
- Remember that small rank differences are often within noise, a point our lesson on how AI is benchmarked develops, and that judges can be biased too, as covered in LLM-as-a-judge.
Our news piece on the leaderboard paper covers the specific results.
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (Chiang et al., 2024)
Leaderboard Hacking: Preference-Based Model Evaluations are Vulnerable to Manipulation (2026)
Arrow's impossibility theorem (overview)
Gibbard-Satterthwaite theorem (overview)
Key questions
What is the Borda count?
Can any voting rule be made manipulation-proof?
How does this apply to AI leaderboards?
Cite this
APA
Ground Truth. (2026, November 13). Borda count and manipulable rankings: why no vote-based leaderboard is gaming-proof. Ground Truth. https://groundtruth.day/learn/borda-count-and-why-ranking-by-votes-can-be-gamed.html
BibTeX
@misc{groundtruth:borda-count-and-why-ranking-by-votes-can-be-gamed,
title = {Borda count and manipulable rankings: why no vote-based leaderboard is gaming-proof},
author = {{Ground Truth}},
year = {2026},
month = {nov},
url = {https://groundtruth.day/learn/borda-count-and-why-ranking-by-votes-can-be-gamed.html}
}