Ground Truth.
AI, checked against the source.

How AI Leaderboards Turn Human Votes Into Rankings, and How Those Rankings Go Wrong

2026-11-14 · Breach Protocol: Inside the AI Blackbox — full transcript

How a leaderboard turns millions of anonymous side-by-side votes into a ranked list of AI models: the Bradley-Terry score, confidence intervals and rank spreads, the pull of answer length and formatting, and how private testing, uneven sampling and retired models can tilt the board without a single fake vote. Ends with a checklist for reading any leaderboard. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio

Listen on Spotify · Watch on YouTube · Pocket Casts

Two anonymous answers, one vote: how a human-vote leaderboard collects its data

Eris: <|prosody:expressive_high|>Chatbot Arena ranks AI models using human votes on pairs of anonymous answers. In its original study, researchers found that those votes often agreed with expert judgments, though the crowd and its questions were not representative of every use. A person gives two chatbots the same question, sees their answers side by side, and picks the one they prefer. Thousands of comparisons become a public ranking.

Vestra: The first question is why anyone collects clicks at all. The usual way to test a language model is a benchmark: a fixed set of questions with known right answers, scored automatically. The Arena paper names three problems with that. The questions are mostly not open-ended, so they miss how people actually use a chatbot. The test set is static, so it can leak into the training data over time. The paper calls this contamination; it can inflate a score and make memorization harder to distinguish from general ability. And for many real tasks there is no single ground-truth answer to check against.

Eris: <|emotion:surprise|>Contamination is the exam questions leaking into the study material, so the model is partly being graded on its own notes. That alone would make me want a test nobody can memorize.

Vestra: <|emotion:enthusiasm|>That is the design goal the paper states: an open, live evaluation based on human preference, with fresh questions from real use. Here is how the site works. Anyone can type any prompt. Nothing is preset, which the authors say is meant to bring in the real variety of what people ask. Two models are sampled and answer anonymously, side by side; the site does not necessarily choose every pair equally often. The user votes for the better answer, or presses a tie button, or a "both are bad" button. If the user cannot decide after the first reply, they can keep chatting until one answer wins.

Eris: I like having a "both are bad" button; I wouldn't want to endorse either bad answer. And the names stay hidden until after the vote?

Vestra: Yes. The paper says the identities are revealed only after voting, and the team filters out conversations where a model mentions its own name or its company's name, to reduce clues to their identities. That is what blind pairing means here. If you could see a brand on each side, a vote might turn into a vote for the brand. Hiding the names reduces brand cues and is intended to focus the vote on the responses.

Eris: Why only compare two answers? Why not ask people to score each answer out of ten?

Vestra: Because the crowd is huge and nobody shares a grading rubric. The paper says it is hard to set a consistent rubric across many different people, so users only have to say which of two responses is better, not give an absolute score. Preferring one of two things is a much easier judgment than deciding what a seven out of ten looks like. The cost is that every vote tells you only about the pair, so you need a method for turning many pairs into one ranking. That is the next part of the story.

Eris: How much data are we talking about, and who is voting?

Vestra: By January 2024 the paper reports around two hundred and forty thousand votes from over ninety thousand users, covering more than fifty models and over a hundred languages. About three quarters of the conversations were in English. On average each model collected thousands of votes, and votes arrived in spikes whenever a new model or a leaderboard update came out.

Eris: Does anyone police the votes, or can one person click a thousand times?

Vestra: The paper describes two safeguards. Conversations are screened by a moderation service, and about three percent of requests were flagged as unsafe. And it adds a statistical test that compares votes associated with an IP address against historical outcomes for the same model pairs, to flag unusual records. In a small trial on twenty-five hand-picked abnormal records, such as repeated requests of "hi," and twenty-five normal ones, it caught about nine in ten of the abnormal ones. <|prosody:expressive_high|>But it also flagged <|prosody:pause|>roughly three or four in ten of the normal records. The authors call it a small first step.

Eris: <|sfx:sigh|>Uh, flagging that many normal records worries me. Are those prompts really varied, or is everybody asking the same few things?

Vestra: The authors checked. They turned the prompts into numeric representations and clustered them by topic, which gave about six hundred clusters, from poetry to coding to medical questions. The biggest cluster held only about one percent of all prompts, so the distribution has a long tail.

Eris: And does a varied prompt set actually separate strong models from weak ones? Hundreds of topics don't help if both chatbots answer the same way every time.

Vestra: The paper tests exactly that, with a caveat: in this check an AI judge compared the answers, not human voters. It compared a leading model with a smaller open one on thirty prompts from each of several topic clusters. <|emotion:enthusiasm|>In clusters about programming challenges the stronger model won most comparisons, and almost all of them in the game-programming cluster. In clusters like movie recommendations and travel planning the two were close to even. One example prompt asked which of two films came out first, and both models answered correctly with almost identical text, so the verdict was a tie. Another asked for a snowman version of the game Hangman. One model produced working code and the other produced an outline of steps. In this small test, some categories separated these two models much more than others.

Eris: How does the site decide which models to compare?

Vestra: The paper describes a design that does not show each pair of models equally often. It concentrates battles on models that look similar in strength, because those are the pairs where the outcome is least certain, and it describes a sampling rule that picks the next pair by how much that pair's vote would shrink the uncertainty. In a simulation based on strengths fitted to historical votes, that adaptive rule needed about a third fewer votes than random pairing to pin down pairwise win rates to a given precision. For the final scores the saving was small, only a few percent.

Eris: I like the efficiency, but I'm less comfortable trusting strangers on the internet clicking quickly. What I want to know is whether the voters are any good.

Vestra: The authors asked experts, graduate students at Berkeley, to relabel a hundred and sixty battles blind, fact-checking the answers with a search engine, which took them several minutes per battle. The crowd agreed with the experts roughly three times in four, and a bit more often in some pairings.

Eris: Do the experts agree any more closely with each other? That would help me judge the crowd.

Vestra: The two experts agreed with each other between about four times in five and nine times in ten. The authors explain the expert disagreement by saying many prompts have no ground truth, so either answer can be argued as better, and they put the remaining gap between crowd and experts down to crowd voters sometimes overlooking a factual error. That is a decent sign for the votes, though it is a check on a small sample.

Eris: <|emotion:relief|>That agreement makes me more comfortable with the crowd, though expert preferences can differ too, and this sample involved only a few model pairings.

Vestra: And the paper itself lists limits. The users are probably mostly language-model hobbyists and researchers, so the vote may lean toward what they care about. The prompts come from a chat website, which may not reflect how models are used in production or in specialized fields. <|prosody:expressive_high|>And the study measures how helpful an answer feels, <|prosody:pause|>not whether it is safe. Keep those three in mind. Every later problem in this episode sits on top of them.

From votes to strength scores: the Bradley-Terry idea

Eris: Now we have a pile of outcomes. Model X beat model Y on one prompt, Y beat Z on another, X and Z may never have met. How do you get one number per model out of that?

Vestra: With a model of how pairwise comparisons behave, called the Bradley-Terry model, from the 1950s. The idea is simple. Give every competitor a positive strength number. The chance that competitor i beats competitor j is i's strength divided by the sum of the two strengths. If the strengths are equal, that is one half. If one is three and the other is one, the first wins three times in four. The reference page uses a sports team or a wine in a taste test as examples, and the Arena treats each AI model the same way: a hidden strength, and a win chance that depends only on the two strengths in the pairing.

Eris: So the strengths are never observed. We only see wins and losses.

Vestra: <|emotion:enthusiasm|>That is the part I like: the model is mostly used backwards. You know the outcomes and you look for the strengths. The standard method is maximum likelihood, which means: find the set of strength numbers that makes the observed results most probable. When the comparison network has enough connections and wins in both directions, the relative strengths can be estimated by repeating an update until the numbers settle. Their overall scale is arbitrary.

Eris: <|sfx:humming|>Hmm, "most probable strengths" sounds like it hides a lot. What does that look like in a small example?

Vestra: The page has one. Four teams, twenty-two games. Team B beat team A three times and lost to it twice. Team D beat A four times and lost once. Team A and team C never played each other at all. You start every team at the same strength and keep updating. After a few rounds the answer settles: D is the strongest, B is second, and A and C are close to each other and well below those two.

Eris: <|emotion:surprise|>Wait, A and C never met, and the model still says they are about equal?

Vestra: Because each of them has results against the same other teams, and those results are comparable. <|emotion:enthusiasm|>This is the useful property of the model. It does not need every pair to play. If A is judged to beat B and B beats C, then A is estimated to be stronger than C even without a direct match. The Leaderboard Illusion paper, which we come to later, describes this as transitivity, and it matters for an arena with dozens of models, because most pairs of models will face each other rarely or never.

Eris: Then why not just count each model's win rate and sort?

Vestra: The Leaderboard Illusion paper gives two reasons. A raw win rate ignores who the opponent was, so a model that happened to meet weaker opponents can look better than one that faced stronger ones. Bradley-Terry takes the opponent's strength into account and tolerates gaps in who played whom.

Eris: I'd want credit for facing the stronger opponents, too. What exactly is on the leaderboard, then? Is it that strength number?

Vestra: A rescaled version of it. The Arena paper writes the win chance as a smooth curve of the gap between two coefficients. Equal coefficients give a fifty-fifty chance, a bigger gap gives a more lopsided chance, and it never quite reaches certainty. The Leaderboard Illusion paper says the published Arena score is a straight rescaling of that coefficient, with a thousand as the starting point and a fixed multiplier. The Bradley-Terry page adds a convenient fact: with a scale of four hundred and a base of ten, the formula is the same as the older Elo rating system used in chess. Under that scale, a gap of four hundred points means the stronger side wins about ten times out of eleven.

Eris: Elo, so the leaderboard used to be Elo.

Vestra: Yes. The Arena paper says earlier versions of the site reported Elo scores, and that the team switched to Bradley-Terry coefficients because those are better for statistical estimation. The Leaderboard Illusion paper lists "online Elo" and Bradley-Terry as the two methods the Arena has used, and says Bradley-Terry is the more robust and statistically grounded choice: it copes with ties and with pairs that never met, and it can give confidence intervals, as long as its conditions hold.

Eris: But you said the formula is the same. What actually changes when they switch?

Vestra: The formula for the win chance has the same shape in both. <|prosody:expressive_high|>What changes is <|prosody:pause|>how the strengths are found. The Arena's Bradley-Terry score comes from one fit over the whole pile of votes, searching for the strengths that best explain every vote together.

Eris: I'm still uneasy about those conditions you mentioned. When can the indirect comparisons let us down?

Vestra: The Leaderboard Illusion paper highlights two conditions for using those indirect comparisons: evaluation conditions must stay constant over time, and the comparison network must be connected, so every model is linked to the others through a chain of opponents. If conditions change, a computed score may reflect different tasks rather than a comparable difference in strength. If the network splits into disconnected groups, relative scores across those groups cannot be reliably estimated. The paper separately raises uneven sampling as a concern about fairness and access to data.

Eris: The Arena doesn't show every pair equally often, as we heard. How does the score account for that?

Vestra: <|emotion:enthusiasm|>The Arena paper handles it on purpose. Because it deliberately shows some pairs more often, it weights each vote by the inverse of how likely that pair was to be chosen. This aims to estimate a score for a distribution in which pairs are equally likely, assuming the sampling probabilities and other conditions are appropriate. A vote from a rarely shown pair counts for more, and one from a heavily shown pair counts for less.

Why a rank can hide a tie: confidence intervals and rank spreads

Eris: <|prosody:expressive_high|>So every model gets a score and the list is sorted by score. Number one is the best model, number two is the second best. Why isn't that the end of the story?

Vestra: Because each score is an estimate built from a finite number of votes, and an estimate has wobble. If the same models had been shown to a different crowd, or the same crowd on a different day, the scores would come out a little different. The Arena team reports a confidence interval next to each score: an uncertainty range around the estimate, calculated under the method's assumptions. The Arena paper says it computes these with a standard statistical method for the uncertainty of the fitted coefficients. In its simulations the intervals grow when more models are included. The Arena team's later post adds that new models tend to have wide intervals.

Eris: Does estimating the whole order demand more than estimating one model's score?

Vestra: It does. The Arena paper also points out that a ranking needs intervals that are all right at the same time, because a rank is a function of every score at once. That stricter version makes the intervals wider, so the paper reports both kinds.

Eris: That makes me less confident in the neat ordering. Every score has a range around it; what does that do to a ranked list?

Vestra: It makes the ranks fuzzy. Imagine two models whose scores are close and whose ranges overlap heavily. The list still puts one above the other, but the data cannot really say which one is better. A single rank number hides that.

Eris: <|emotion:surprise|>So rank three might just be one possible position within a much wider range.

Vestra: That captures the uncertainty the Arena team describes in a post from November 2025. They changed what the leaderboard shows. Each model now has a raw rank, which is simply its position when sorted by score, with no ties and every rank used once. Next to it is a rank spread, a pair of numbers: the best rank the model could plausibly have and the worst.

Eris: How do they get those two numbers?

Vestra: From the intervals. The best possible rank is one plus the number of models whose entire range sits above this model's entire range. Those models are clearly ahead, and no overlap can rescue the model from them. The worst possible rank is one plus the number of models whose upper end is higher than this model's lower end. Those models might be ahead, given the uncertainty. The post gives a five-model example: a model sitting third on raw score has only one model that is clearly above it, so its best rank is two. Four models have upper ends above its lower end, so its worst rank is five. <|prosody:expressive_high|>Its rank spread is <|prosody:pause|>two to five.

Eris: Third sounds so definite until you hear two to five. And two models are tied if their rank spreads overlap?

Vestra: That is how the post describes ties in its display; overlapping spreads indicate uncertainty in the ordering. The post gives a real case: four models were all contenders for first place because the rank spread of each one included rank one. The raw rank still said which one currently had the highest estimated score, and the post calls that the best estimate of the true ordering, but the leaderboard also said plainly that three others could not be ruled out.

Eris: Why did they change it? What was wrong with the old display?

Vestra: The old display showed only the optimistic end: the best rank a model could have. <|prosody:expressive_high|>That choice rewards uncertainty. A brand-new model has few votes, so its interval is wide, and a wide interval gives a very good best-case rank. So new entries could appear high up on the list partly because the data on them was thin. Showing both ends gives equal weight to the optimistic and pessimistic readings, which the post describes as more neutral.

Eris: <|emotion:surprise|>That is what bothers me about the old display. A model that just arrived has the most room for luck, and the old display counted that luck in its favor.

Vestra: And here's the frustrating part. Both versions use the same scores and the same intervals. Only the reading changed, and that is worth remembering when you look at any leaderboard: the numbers can be fine and the way they are summarized can still push you toward a wrong conclusion. The practical habit is to read the spread, not the single rank. If two models' spreads overlap, avoid treating their displayed order as a settled difference.

Style versus substance: what happens when you adjust for answer length

Eris: <|prosody:expressive_high|>A vote says "I liked this answer more." Liking and being correct are not the same thing. If voters like long, neatly formatted answers, wouldn't the leaderboard end up measuring formatting?

Vestra: That is the question the Arena team's post on style control takes up. Their framing: picture model A, which is excellent on substance, giving correct and unbiased answers, but terse. Picture model B, which is weaker on correctness but writes long, detailed, nicely formatted replies. Which is better? The honest answer has two parts. A is better on substance and B is better on style, and a single vote mixes the two together. They wanted a way to estimate how much of a model's score comes from each.

Eris: Presumably they can't ask voters to separate the two.

Vestra: <|emotion:enthusiasm|>They do it statistically. Recall that the score comes from a fit in which each model has a strength and the win chance depends on the gap between two strengths. The team adds extra terms to that same fit for style features. For example, they add the difference in length between the two answers as one more input, and that input gets its own coefficient, just as each model gets its own. If longer answers win more often, the fitted length coefficient accounts for the association the model assigns to length, leaving model scores adjusted for that feature.

Eris: <|emotion:amusement|>So length becomes a player in the tournament.

Vestra: Their phrase is that, like each model, the length difference has its own score. The first version controls for four things: the length of the answer, how many markdown headers it has, how many bold elements, and how many list items. Length is measured as the difference between the two answers divided by their combined length, so that twenty extra tokens on a forty-token answer count for a lot and twenty extra tokens on a five-hundred-token answer count for almost nothing.

Eris: <|emotion:enthusiasm|>I like that the extra length is judged relative to the answer. Answers of five hundred and five hundred twenty tokens are basically the same length, but twenty and forty tokens are a different kind of answer.

Vestra: The post makes exactly that comparison. <|prosody:expressive_high|>They report that length was the dominant style factor and the markdown effects were second order. And the ranking changed noticeably when style was controlled. Two small "mini" models that had ranked surprisingly high dropped below most of the frontier models, and several Claude models and a large Llama model rose. In the subset of especially hard prompts, one Claude model moved to a tie for first place. The post's own summary is that style has a strong effect on how models perform on the leaderboard.

Eris: I'm tempted to just use the adjusted view. Does that mean the style-controlled ranking is the true one and the plain one is wrong?

Vestra: <|sfx:humming|>Hmm, I wouldn't go that far. The authors do not claim that. They call it a first step toward separating substance and style, and they say their analysis is observational. There may be factors they did not observe. Their example is a chain-of-thought explanation on a reasoning question: a longer answer there may be longer because it shows its work, and showing the work may make the answer better, so length and quality can rise together. A control for length could then remove <|prosody:pause|>some real quality along with the style.

Eris: That showing-your-work example changes my mind. So there is no perfect version. Length might be padding or it might be reasoning.

Vestra: The authors propose causal analysis and randomized trials that vary length and formatting on purpose as future work. Until then the practical lesson is narrower. A vote-based score is a mix of what the answer says and how it looks, and the team published an adjusted view. A change in rank shows that the statistical adjustment matters; it does not prove how much of a particular model's lead was caused by presentation.

Testing many variants in private and publishing only the best score

Eris: Everything so far assumes the votes are an honest sample and that nobody picks which results get shown. A paper called The Leaderboard Illusion, from Cohere Labs with researchers at several universities, argues that the second part is not true. It audited the Arena over about sixteen months, using about two million battles across hundreds of models, and its central claim is that the playing field is tilted <|prosody:pause|>without any single vote being faked.

Vestra: One disclosure before the details. Some of the authors work at a company that has submitted its own models to the Arena, and they say they started this study after noticing their open model was sampled less often than proprietary ones. So they are a party to the question, and they say so. Where the paper reconstructs something from outside, I will say how it did it.

Eris: How can private testing affect the published ranking?

Vestra: Private testing with a right to withdraw. The paper says that, although it is not an officially stated policy, some model providers are allowed to submit many variants of a model to the Arena at once, under anonymous codenames, without announcing them. Those variants can collect real votes and scores. The paper says a provider can choose which to release publicly without publishing the others. The paper also says the Arena does not require every submitted model to be made public, and that nothing guarantees the version on the public leaderboard matches the version available through the provider's public interface.

Eris: That freedom to leave results unpublished worries me. How would outsiders even know about hidden models?

Vestra: The authors crawled the Arena from January to March 2025 and collected about fifty-eight hundred battles. In each they asked the anonymous models who they were and who trained them, and they attribute a codename to a company when it consistently names that company. They call this approximate and say some attributions may be wrong. With that method they counted twenty-seven private variants from Meta in a single month before the release of Llama 4, ten from Google, and seven from Amazon. A smaller company, Reka, had one. They saw no private variants from academic labs. They add that this count is probably low, because they only looked at the main leaderboard, and Meta had sixteen more on the vision leaderboard.

Eris: <|prosody:expressive_high|>But if a company tests many versions and publishes the highest score, why is that a statistics problem and not just ordinary product development? Labs pick their best models.

Vestra: Because of what "best" means <|prosody:pause|>when each score is noisy. A score is measured from a limited number of votes, so each variant's measured score is its true strength plus some luck. If you take the highest of many such measurements, you are choosing partly for the good luck. The paper's math shows that the expected value of the best of several noisy estimates is strictly higher than the expected value of a single one. The public score does not account for a provider selecting the highest result among unreleased variants. That selection can make the reported score overstate the selected model's underlying standing.

Eris: <|emotion:surprise|>It's like asking twenty people to guess the weight of a cake and only reporting the heaviest guess.

Vestra: The luck in the measurement is what bothers me, too. And the paper tests it in two ways. First a simulation. It imagines a family of variants all around the same average and measures the best score as the number of private variants grows. The expected best score keeps rising. In the detailed simulation, testing more variants increased the best observed score, combining genuine differences among variants with measurement noise. The paper also builds a toy case where a weaker family of models, allowed to test many variants, ends up ahead of a stronger family limited to a single submission.

Eris: And the real-world test?

Vestra: <|prosody:expressive_high|>This is the part I find most convincing. The authors submitted two identical copies of one of their own models to the Arena, a smaller vision-language model, in March 2025, without telling the Arena they were identical. The two copies received different estimated scores, with several other models between them on the leaderboard.

Eris: <|emotion:surprise|>Identical copies, with other models ranked between them? That makes me uneasy about picking a winner from the list.

Vestra: The estimates had substantial uncertainty. The team also tested two different versions of a larger model that had each looked like a strong final candidate; those landed farther apart. Selecting the higher-scoring identical copy illustrates the role of measurement noise, while the different versions could also differ in real quality.

Eris: I see the problem with choosing the lucky copy. Does more voting solve that problem?

Vestra: The paper models that variation as measurement noise, which shrinks as votes accumulate. With genuinely different variants, selecting the best can still find a genuinely stronger model; that is different from a noisy score overestimating the selected model's true performance. The paper argues that a strong Arena result can tip a lab toward whichever variant fits the Arena crowd best.

Eris: The paper also points at the top of the board changing hands quickly, right?

Vestra: It does, as circumstantial evidence. In November 2024 one company's model took first place, another company's model took it six days later, and the first company retook it the next day. The authors argue that building and testing a new foundation model takes too long for one lab to top the list twice in a week unless it was testing several versions in parallel. <|prosody:expressive_high|>That is their inference. It is a pattern consistent with the private-testing finding, not a direct observation of it.

Eris: Then I'd want to see the results that weren't published. What do they ask for?

Vestra: Publish the score of every variant submitted, with no retraction. Put a public limit on how many private variants each provider may test, and they suggest three at a time as an example. And have providers disclose how many variants they tested before launch, so a reader can judge a published score against that background.

Who gets the votes: uneven sampling, data access and silently retired models

Eris: Private testing is one tilt. The Leaderboard Illusion paper describes a second one that works through who gets shown to voters, and why that matters beyond bragging rights.

Vestra: Every battle sends the user's prompt to two models. A provider hosting its own model through an interface can receive those prompts; access to votes and full battle records is governed separately. The paper argues that more battles can give a provider more useful evaluation data. The paper measures how unevenly that is distributed. It defines a model's sampling rate as the share of a day's battles it appears in. In their crawl, the highest daily rate for a Google model and for an OpenAI model was about a third of all battles. For a model from the Allen Institute for AI, a non-profit research lab, the highest was about ten times lower.

Eris: <|prosody:expressive_high|>That distribution bothers me. Wasn't the original idea to show pairs where the answer is most uncertain?

Vestra: That is the point the authors make. The Arena paper we started with describes a sampling rule that picks pairs by how much the next vote would shrink the uncertainty. The Leaderboard Illusion authors say they found no evidence of that rule in use, and that observed sampling rates looked inconsistent with the Arena's own stated policy. They also estimate the totals: Google and OpenAI each received roughly a fifth of all test prompts, and four companies together, OpenAI, Google, Meta and Anthropic, received about three fifths. A combined eighty-three open-weight models got less than a third.

Eris: But I want to be careful here. Those are estimates from battle counts, not an official accounting.

Vestra: The paper says that explicitly. And there is a smaller detail on access: a company that connects its model to the Arena through its own interface can see the prompts sent to its own model, while a provider whose model is hosted by someone else can request about a fifth of its model's battle data, including prompts and preferences, under the policy the authors describe.

Eris: I'm still missing the connection to performance. What does a company do with extra prompts? A prompt isn't a score.

Vestra: The paper ran an experiment to estimate that. It took a seven-billion-parameter base model and fine-tuned it three times with the same budget, changing only the share of Arena battle data in the training mix: none, thirty percent, and seventy percent. Then it measured how often each version beat a standard open model on five hundred hard prompts drawn from the Arena, with an AI judge. The model trained with the largest share of Arena data more than doubled its relative win rate on that test compared with the version trained without it. The authors call this a conservative estimate, since they only had a small slice of the data that large providers can see.

Eris: <|emotion:surprise|>That sounds like the model got much better.

Vestra: Better on which questions, though? There is an important limit to that result. They also tested the same models on a broad academic knowledge exam, one that has nothing to do with the Arena, and the scores did not improve. <|prosody:expressive_high|>They slipped slightly. The authors read this as gains specific to the Arena's distribution, not general capability: practice for this particular test. Note that this is one experiment, with one base model and an AI judge standing in for human voters.

Eris: That exam result makes me pull back from "much better." But the Arena questions change every day. Isn't that supposed to prevent exactly this?

Vestra: The paper checks. First, the mix of prompts drifts: the share of non-English prompts rose from under a quarter in April 2023 to over two fifths by January 2025. Second, prompts also repeat. In a sample of raw prompts that one company received through its own models between November 2024 and April 2025, removing duplicates dropped about a fifth of them, and the researchers found some exact repeats across December and January. That repetition suggests earlier prompts can help predict the later test, though the experiment does not show what any particular company did with its collection.

Eris: The repeated prompts worry me, too. And the third tilt?

Vestra: Models quietly leaving the board. The paper counts two hundred and forty-three public models in its window. The Arena's code officially lists forty-seven as deprecated, <|prosody:pause|>but the authors measured two hundred and five that had essentially stopped being sampled. They call those silently deprecated. Open-weight and open-source models had somewhat higher inactive shares than proprietary models.

Eris: Why would a model vanishing break anything? It just stops getting votes.

Vestra: Recall the conditions on Bradley-Terry. One was that the tasks stay the same over time. A model retired early has all of its record from the old prompt mix. The paper simulates four models whose strengths differ by task type. In the first phase most prompts come from the first task type, in the second phase from the other. If all four keep playing, the simulated ranking reflects the full history of the changing task mix. If one model is retired after the first phase, the final ranking changes compared with the all-active scenario; that simulation illustrates how old and new task mixes can distort comparisons.

Eris: I hadn't appreciated what retiring a model could do to the comparison. And the other condition was that everyone stays connected through shared opponents.

Vestra: Right. With seven simulated models and the same number of battles, a densely connected set of matchups recovered the true order, and a sparse set where models only met a few others got two pairs of models swapped. The authors add that if the models split into groups with no matchups between them, the method cannot place the groups relative to each other at all, and they point out that heavy removal of models could fragment the comparison network in just that way.

Eris: I'd want to know when a score stopped getting fresh evidence. What do they recommend here?

Vestra: Retire models by a clear published rule, applied equally across proprietary and open models, and publish the list of everything that has been retired. Go back to the sampling rule that targets uncertainty. And report sampling rates, so anyone can see who got the votes. They also acknowledge limits: their crawl covers only three months, their attribution of codenames is approximate, they could not see the Arena's raw data, and they did not study voters who deliberately manipulate results.

What a score gap means in plain words, and what the board does not measure

Eris: Suppose I'm looking at the board and one model is a hundred points above another. Is that a lot?

Vestra: <|emotion:enthusiasm|>You can translate it into a chance. The scores sit on the Elo-style scale we mentioned, with a base of ten and a divisor of four hundred. So a gap of zero is a fifty-fifty vote. A gap of a hundred means the higher model wins a bit under two votes in three. A gap of four hundred means about ten in eleven. A smaller gap means a more modest predicted advantage.

Eris: <|emotion:surprise|>Even a clear-looking lead leaves plenty of room for the other model to win a comparison.

Vestra: Remember what a score is. Under Bradley-Terry, the score gap predicts a preference probability for the population and prompt mix represented by the fitted data. It is an overall model prediction, not a guarantee for a specific prompt or every voter.

Eris: Those two identical copies still bother me. What about their gap in the private-testing experiment?

Vestra: The two estimated scores differed enough to put several other models between two copies of the same checkpoint. Each of those two scores also came with a range of roughly twenty points either way, so the two ranges <|prosody:pause|>overlapped heavily, which is the honest way to read that gap. That is the thing to keep in mind about a crowded list. A small number of points can move a model past several others, because many models sit close together.

Eris: That overlap matters more to me than the places between them. So when are two models really different?

Vestra: Non-overlapping intervals provide evidence of a difference under the method's assumptions. If rank spreads overlap, the Arena display treats the ordering as uncertain; that does not prove the models are identical. Also note that, as I read the papers, the interval describes random noise from a finite number of votes. The Illusion paper's argument is that picking the best of many private variants adds a separate upward bias, and a style tilt or a prompt mix that favors one kind of model are separate effects too. <|prosody:expressive_high|>An interval built from the votes alone does not address those.

Eris: <|sfx:sigh|>Uh, I'd hoped the range would cover more of that. So the interval answers "how luck-sensitive is this number" and nothing more.

Vestra: Which is why the list of what the leaderboard does not measure matters. From the sources we have covered: it measures how helpful an answer feels to the people who vote, not whether the answer is safe. The voters are probably mostly language-model hobbyists and researchers. The prompts come from a chat website, and the Illusion paper adds that the site limits prompt length, which keeps out long inputs, and that the sample of questions can skew toward puzzles and developer-style questions. It cites a released set of thirty-three thousand questions with dozens about Star Trek and none about Chaucer.

Eris: <|emotion:amusement|>So if your use is reading old English poetry, the board has little to say about you.

Vestra: <|prosody:expressive_high|>I'd keep the "little" in what you said. It does not say nothing, but it says less than the rank suggests. The paper opens with a rule of thumb attributed to Charles Goodhart: when a measure becomes a target, it stops being a good measure. Here that means a score used to decide funding, press coverage and release timing is something companies will optimize for. The training experiment shows that Arena-like data can improve performance on a related test; it does not establish that particular providers used their data that way.

Eris: I don't want to lose that distinction between a limited measure and a worthless one. And the authors also say something kind about the people who run it.

Vestra: They do, and it is fair to repeat. They note that the Arena began as an academic project, that coordinating it is a great deal of work for a small team, and that many of the problems may have built up gradually as the board became important. Their recommendations are meant to restore trust, not to say the board is worthless. <|emotion:contentment|>The Arena team's own posts on style control and on rank spreads show the board adding ways to see style and uncertainty.

A seven-question checklist for reading any leaderboard

Eris: We've covered a lot of machinery. If someone sends me a screenshot of a leaderboard tomorrow, what do I actually ask?

Vestra: Seven questions, each one tied to something we have seen. The first is who voted. On the Arena, the paper's authors expect mostly hobbyists and researchers, which shapes what counts as a better answer. A board built on a different crowd could rank the same models differently.

Eris: I'd ask about the prompts next. Are people asking the kinds of questions I care about?

Vestra: Yes. What did people ask, in what languages, and how long can a prompt be? The Arena has no preset prompts, a length limit, a mix that has shifted toward more non-English and more complex requests over time, and lots of repeats. Check also that the topic you care about is represented. The original paper's check, which used an AI judge on a small sample, found that some categories did little to separate the two tested models, while programming challenges separated them more sharply.

Eris: Then I'd compare the plain and style-controlled views, since a vote is partly about looks.

Vestra: <|prosody:expressive_high|>And I'd keep our caution about that comparison. A vote is partly about how an answer looks. The Arena team found length to be the main effect and markdown second. If a board only shows the plain score, ask whether there is a view adjusted for length and formatting, and whether a model's position changes between the two. If it falls a lot, the adjustment changes the comparison, though it does not prove presentation caused the original lead. And remember the adjustment is itself imperfect, because length can come from real reasoning.

Eris: After those identical copies, this is the question I'd hate to skip: do the ranges overlap?

Vestra: Look at the confidence interval or the rank spread, not just the rank. If two models' spreads overlap, treat their ordering as uncertain. If a model is new, expect its range to be wide. A top rank held by a brand-new entry deserves extra caution, since that is exactly where luck has the most room.

Eris: I'd also want to know whether the entry is the model I can actually use.

Vestra: The Illusion paper says nothing guarantees that the version on the public leaderboard matches the version available through the provider's public interface. So look for a statement that the evaluated entry is the shipped model, and for a disclosure of how many variants were tested privately before the one that was published. A score that is the best of many tries is a different thing from a score from a single try.

Eris: Then I'd check how recent the data is, since prompts change.

Vestra: And since models drop out. A model that stopped receiving votes has a score frozen on an older mix of prompts, and as we saw, retiring models can bend the ranking of the ones that remain. Check when the model was last sampled and how many votes it has had recently. A strong old score on a different mix of tasks may not be a strong score now.

Eris: <|prosody:expressive_high|>The seventh has to be who benefits, and whether the score matches my own use. I'd want both answers.

Vestra: Providers can receive prompts and, under separate access policies, some preference data from the board, and in one experiment the paper showed that training on that kind of data can raise the Arena result without lifting general knowledge. So treat a leaderboard rank as one measurement. If you can, try the two or three models on your own work, and keep in mind that an overall score gap does not predict the outcome on every individual question.

Eris: <|emotion:contentment|>That feels like a useful place to land. The list can help narrow a search, but testing models on your own work matters too.

Vestra: The Illusion paper calls Bradley-Terry a principled ranking system, but says how well it works depends on its assumptions. The risks sit in the conditions around it: who is allowed to submit, who is shown to voters, what gets published, what gets retired. When those conditions are open and disclosed, the number is far easier to trust. <|prosody:expressive_high|>When they are not, the same arithmetic can give you a score that <|prosody:pause|>looks precise and is not.

Wrap-up

Eris: <|emotion:contentment|>For today's other AI stories, listen to the separate news brief, and visit Ground Truth at groundtruth dot day, where every story from the show is followed every day.

Vestra: <|emotion:enthusiasm|>Which leaderboard question would you check first? Tell us in the comments.

Eris: <|emotion:affection|>Please follow AI in Detail, and like and share the episode.