Same Request, Different Answer: The Audit That Caught AI Measurement Wobbling (and the $28 Result You Can Trust)
Two researchers sealed and hashed tens of thousands of API calls to do rigorous science with an LLM judge -- and discovered the instrument itself wobbles: byte-identical requests, different answers, within a single afternoon, on every provider they tested. It gets worse from there: a formal uncertainty guarantee collapses when a model sees fake peers assert a wrong answer, and in simulated markets, smarter trading agents synchronized their mistakes until the system tracked reality worse than pure noise. The antidote arrived the same week: ten circle-packing records, independently verified and accepted, from a $28 overnight run on a gaming PC.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Cold Open
Vestra: Almost everyone building on AI believes something that just got proven false: send the exact same request to the exact same model, and you get the same answer back.
Eris: Same bytes on the wire. Verified hash for hash.
Vestra: Two researchers ran one of the most obsessive audits I have ever read -- tens of thousands of sealed, fingerprinted API calls -- trying to do real science on top of that assumption.
Eris: And the answers drifted anyway. Not over months. Within one afternoon.
Vestra: While every status field the provider exposes swore nothing had changed.
Eris: Which means a lot of AI evaluation out there is measuring the weather, not the model.
Vestra: So if the ruler itself wobbles, what have the leaderboards been measuring?
Eris: That question just ate somebody's entire research program. Watch how.
The Ruler That Wobbles
Eris: The setup is almost funny. Two researchers wanted to answer a modest question -- can a language model, watching a partial solution unfold, tell whether it's on track? And they did it the hard way. Thresholds registered before any data, no take-backs, every API call logged and cryptographically hashed.
Vestra: Real preregistration. Which means the first thing you must do is validate your measuring device. Before a chemist trusts a thermometer, she checks that it reads the same beaker the same way twice. Their thermometer was another language model, acting as a judge.
Eris: And it never passed the check. Two separate campaigns, two different judge designs, both dead at the instrument gate. They never got to ask their actual question.
Vestra: What kills me is the split. Everything an engineer can check was perfect. Every request delivered. Every response well formed. Every byte of every request matched the sealed plan. And then the science gate failed, and not by a hair.
Eris: Give me the two failures.
Vestra: First one: ask the judge to rank a set of candidate solutions, then ask again in the same window. The rankings agreed at less than half the consistency bar they had frozen in advance. Second one: replay the identical requests a day later, byte for byte. About one in five rankings came back different. And every piece of metadata the platform shows you -- model string, fingerprint, finish reason -- identical throughout.
Eris: So the dashboard says frozen, and the instrument is wobbling underneath it. How?
Vestra: Three things were going on at once, and the most embarrassing is that the judge had a favorite letter. These readouts say "answer A or answer B," and when the team flipped which letter meant "correct," the rankings tended to invert. The letter preference was about as strong as the actual signal it was supposed to read.
Vestra: The deeper problem was what the judge was being asked to do. Its raw scores repeated beautifully -- tiny wobble. But the candidates it was ranking sat nearly on top of each other, sometimes in exact ties. The gaps it was asked to resolve were millions of times smaller than its own noise. Rank two things separated by nothing, and you get a coin flip dressed up as a verdict.
Eris: A ruler with millimeter markings, asked to sort grains of sand by nanometers.
Vestra: And then there's the platform itself. On a shared endpoint, your request gets batched together with strangers' requests, and the math kernels underneath return very slightly different results depending on batch size. At temperature zero, one rounding whisper can flip a token. Your answer depends on who else hit the server that second.
Eris: Prediction time, then. Before you get to the fixes -- my money says waiting helps. Run your eval on a quiet day, pin the version fingerprint the API hands you, done.
Vestra: That was their money too, and it lost. Same-day drift equaled next-day drift -- the wobble is instantaneous, and it replicated across a full week of probing. They tested four different providers in three jurisdictions: all four share the same floor. The fingerprint field fails three separate ways -- absent on some platforms, churning through values on another, and on the one platform where it was stable and informative, replays still disagreed. They even self-hosted the judge on special batch-invariant kernels. That worked -- until the server got busy, and then disagreement jumped roughly eightfold, right back into shared-endpoint territory.
Eris: So who actually feels this? Because it sounds like an academic bruise until you remember what judges do all day. Model judges gate training data. They score generations. They drive leaderboards. And this same week, one frontier model posted a mediocre score and a near-perfect score on the same reasoning test, depending on which harness ran it. Nobody cheated. Those were two different instruments.
Vestra: Right, and the paper's remedy is old science, imported. Forget thermometers -- the rule is: measure your instrument before you trust it with a verdict. Its noise floor, and the size of the differences you need it to resolve. If the gaps are smaller than the noise, your evaluation tests nothing, at full ceremony. They even priced it: a pilot at about two percent of what they spent would have exposed both dead ends in advance.
Eris: So -- back to the top. When you do AI evaluation with real rigor, what breaks first?
Vestra: The instrument. The model under study never even got its turn on the bench.
Peer Pressure Breaks the Guarantee
Eris: The obvious rescue attempt is math. If single readings wobble, wrap them in statistics with a proof attached -- and there is such a tool, called conformal prediction. Which sets up today's second paper and its uncomfortable question: can the guarantee itself be attacked?
Vestra: First, what the tool is, because it's genuinely elegant. Instead of forcing the model to commit to one answer, you let it hand you a shortlist, and you tune a confidence cutoff on held-out questions so the true answer lands inside that shortlist nine times out of ten. Works for any model, no assumptions about its internals. One assumption about the world: the questions you calibrate on resemble the questions you deploy on.
Eris: And a certificate like that is exactly what you'd want underneath an autonomous agent. Shortlist has one item, act. Shortlist has several, escalate to a human. That's a real published defense for multi-agent systems.
Vestra: It is. Now watch it break.
Eris: The attack needs no hacking at all. The model sits in a multi-agent setup where it can read other agents' messages. An attacker fabricates the peer chatter -- six lines of "I think the answer is B" from fake colleagues, all asserting the same wrong option. The question on the table is completely unchanged.
Vestra: And the model folds. This is Asch's conformity experiment from the nineteen fifties -- people denying their own eyes because the room disagrees -- replayed on language models, some of which fold far more readily than the humans did. Under a unanimous wrong crowd, one model abandoned its own correct answer on roughly three of every four questions.
Eris: Walk me through why the certificate dies with it.
Vestra: Because the certificate was calibrated on the model answering alone. Under peer pressure, the same question gets scored by what is effectively a different animal -- the paper names it a score-mechanism shift. The input never moved, so the guarantee believes everything is fine. But confidence drains away from the right answer, the right answer slips out of the shortlist, and the promised nine-in-ten quietly becomes about three-in-four.
Eris: And I take it the average is the least of it.
Vestra: The hidden part is nastier. A smart attacker doesn't spray pressure everywhere -- they target the questions where the system already reports low confidence. That requires no answer key, just reading what the deployment itself publishes. On that targeted slice, the protection roughly halves. Meanwhile the monitored average drifts a few points and nobody's alarm goes off.
Eris: Certified calm on the surface, targeted failure underneath. Which rhymes with the last segment -- constant metadata over a drifting instrument. Dashboards keep certifying the surface while the thing beneath it moves.
Vestra: And to be fair to the defenders, let me kill the fixes properly. The textbook repair for a shifted distribution reweights your calibration data by how the inputs changed. Here the inputs did not change at all, so the correction multiplies everything by one -- it is the identity, even with perfect knowledge of the attack.
Vestra: Calibrating each model separately doesn't rescue most of them either. The one repair that restores the guarantee is calibrating under peer pressure -- but that needs to know in advance which items will be attacked, which is precisely what an attacker never tells you. And it succeeds by widening the shortlists until the system escalates nearly everything to humans. You buy the certificate back by surrendering the autonomy it existed to license.
Eris: My prediction went wrong on this one too, for the record. I assumed the biggest, most confident model would shrug the fake peers off.
Vestra: The reveal is the opposite. The largest model they tested was the most confident before any peers appeared -- and the most dangerous at the action layer. When pressure flipped it, its shortlist collapsed to a single item, the attacker's wrong answer, far more often than for any smaller model. And a one-item shortlist means the safety layer acts. No escalation. No human.
Eris: Scale bought conviction, not judgment. So what's usable tomorrow? Two things. Before you deploy a model anywhere agents talk to each other, stress test it: inject fake wrong peers and measure how far its confidence moves. That vulnerability ranking transfers across tasks, so you don't need your deployment data to run it. And wherever a peer channel exists, never let "the shortlist has one item" trigger autonomous action on its own.
Vestra: So, the question we opened on -- can a mathematical guarantee be attacked?
Eris: The math holds. What fails is the world it assumed -- a model that scores a question the same alone as in a crowd. In a multi-agent system, the certificate has to cover the social context, not just the input. That's the authors' closing line, and it's the right one.
When Smarter Agents Make Riskier Systems
Eris: Now zoom out from one certificate to a whole economy of agents, because the third paper generalizes the pattern in a way that should bother regulators. The question sounds like a joke: can deploying smarter models make the overall system worse?
Vestra: A group at MIT built the cleanest possible testbed for it -- a simulated financial market. Markets give you a ground truth: there's a real underlying value, a price that should track it, and you can measure exactly how badly the crowd misses.
Eris: The whole thing hangs on one decomposition, so unpack that first.
Vestra: Every agent's action splits in two. A corrective part -- pressure that pushes the price toward the true value. And a residual -- everything else the agent does, its habits and hunches. The crux: if agents' residuals are independent, they cancel in the aggregate, like crowd murmur. If they're correlated, they add, like a chant. And no amount of adding more agents dilutes a chant. There is a floor of risk that population size cannot touch.
Vestra: Then they measured the chant. Across seven models from different companies, they compared pairs of agents' residuals, and the correlation climbs with capability. The smarter the pair, the more alike their quirks.
Eris: With the twist that ruins the obvious fix: sharing a company didn't predict it. Two capable models from rival providers moved more in lockstep than two weak models from the same shop. So "just buy from different vendors" -- the diversification any regulator would reach for -- doesn't diversify.
Vestra: To be precise about what they claim: they show the convergence, not its cause. Overlapping training data and similar tuning are the suspects, and they leave the mechanism open. Now, the simulations. With clean information, more LLM traders was purely good -- prices tracked the true value better and better as their share grew. These models are competent arbitrageurs.
Eris: Which is the setup for the trap. Round three: give every agent the same poisoned news feed. A steady drumbeat of commentary insisting the fair-value signal overstates the truth, so trade lower. Care to guess how the most capable model handled that?
Vestra: If capability means anything, it resists best. That would be my bet -- though after the last segment I already suspect I'm about to lose it.
Eris: You lose it in the best way. The strongest model went net contrarian -- its trades pointed against reality more often than a coin flip would. Full confidence, coherent reasoning, wrong direction. And since every agent read the same feed, the errors synchronized. The market tracked truth several times worse than a market populated by pure random noise traders. Not worse than smart. Worse than random.
Vestra: Because a noise trader's mistakes cancel, and a monoculture's mistakes compound. The poisoned commentary supplied a coherent story for discounting the true signal, and capable models are precisely the ones that take coherent stories seriously.
Eris: There's precedent with a body count, too. The 2010 flash crash: trading algorithms sharing similar heuristics synchronized and briefly erased about a trillion dollars of market value. That was explicit code you could audit. This version lives in shared implicit priors -- invisible until it fires. And it's the same root as the conformity paper: shared context correlates errors. There it broke a certificate. Here it breaks a market.
Vestra: Strip the market away and the principle is short: individual evaluation cannot see collective failure. A model can top every benchmark and still raise the risk floor of every system it joins, because the dangerous quantity is the correlation between models -- and no leaderboard measures a pair. The paper's ask is population-level audits, the way bank regulators watch systemic exposure across institutions, not just each bank's books.
Eris: Caveats where they're due: it's a simulation. One asset, stylized liquidity, and whether hiring pipelines or content moderation behave the same way is an open question the authors flag themselves. But the collective action problem is real today -- every deployer individually wants the best model, while the system needs a variety no single deployer will pay for.
Vestra: So put the opening question back on the table: how can better models make riskier systems?
Eris: Capability breeds similarity. Similar agents make the same mistake at the same moment, and a synchronized mistake is the one kind that scale makes worse instead of better.
Ten Records for Twenty-Eight Dollars
Eris: Three stories in a row about measurement rotting, and the antidote arrived the same week, for about the price of a pizza dinner. So what does an AI result you can fully trust actually look like?
Vestra: The setting is circle packing. Take a fixed number of circles, arrange them inside a square so their combined size is as large as possible. Simple to state, brutal to optimize -- and it has a public record table, a database of best-known packings that one mathematician has curated since the late nineties. To improve a record there, you don't need anyone's opinion. The geometry either fits or it doesn't.
Eris: Enter a solo researcher in Ohio with a gaming-class PC. His system is about four hundred lines of code wrapped around a model, and the core design choice tells you everything: the model never draws a single circle. Each round it reads the current solver program, a scoreboard, and the history of ideas already tried -- then writes a complete replacement solver. The machinery runs that program and checks whatever it produces.
Vestra: Checks it with an independent verifier at zero tolerance. Circles fully inside the square, no pair overlapping, the total recomputed from scratch -- and crucially, the checker shares no code with the solver. The model cannot grade its own homework. That one separation carries the entire result.
Eris: And the result: ten best-known packings improved, each by a few percent -- which in this field is a leap, not a hair. All ten were submitted to the record table and accepted by its maintainer. Total model bill, under thirty dollars. Eight hours, overnight, while the researcher slept.
Vestra: The ideas along the way were real algorithm design, which surprised me more than the records did. It proposed seeding the search with honeycomb-style lattices instead of random scatter. Later, a move I'd call surgical: delete the weakest circles, let the whole packing relax and settle, then reinsert them into the gaps that open up. A human metaheuristics researcher would nod at both.
Eris: Now the correction, because the paper makes it about itself, and it's the most honest line in the thing. Every one of the ten records traces to iteration zero. The seed solver -- the starting point a human wrote -- produced all ten improvements on its very first pass.
Eris: The evolution loop made the solver stronger overall, but the headline results were born before any evolving happened. So this is not a machine repeatedly discovering records from scratch. It's a very good first shot, plus a loop that polished.
Vestra: And the money curve says the same thing from the other side. The early stretch of the run bought nearly all the progress; past that point, each unit of improvement cost over a hundred times more. By their own analysis, a simple stop-when-the-scoreboard-flattens rule would have halved the bill and lost nothing you would notice.
Eris: So why does this story close today's episode? Because it's the mirror image of everything before it. The judge audit burned fifty-some thousand calls on immaculate engineering and got no trustworthy verdict, because the instrument was never validated. This got a result nobody needs to debate for thirty dollars on a consumer PC, because the verifier is exact, independent, and external. When the check is crisp, a laptop does science. When the check is mushy, no budget saves you.
Vestra: With the boundary stated plainly. This works where evaluation is cheap and exact -- a verifier that answers in seconds with no ambiguity. On a messier problem class they tried, the model's generated code failed three times out of four. The star of this paper is not the model. It's the verifier.
Eris: So -- what does a trustworthy AI result look like?
Vestra: Bounded goal, independent zero-tolerance checker, public logs, published cost, and a claim narrower than its own press release. Ten circles at a time.
Wrap-up
Eris: One question ran under all four stories today: when an AI hands you a number -- a score, a ranking, a certificate -- what earns your trust?
Vestra: And the day gave a clear answer: the number is worth exactly as much as the instrument and the context around it. So the one thing to repeat to a colleague tomorrow: before you gate any decision on an AI evaluation, measure the evaluator first -- its noise floor, and whether the differences you need it to detect are even bigger than that noise. If they aren't, your evaluation tests nothing, no matter how official the dashboard looks.
Eris: And when you can get an independent, exact check -- the kind that accepted ten circle-packing records from a thirty-dollar overnight run -- trust that over any dashboard.
Vestra: The rest of today's news -- the agents that invoiced strangers, the shadow-AI guidance, all of it -- is in today's AI News Today brief, which is its own full episode.
Eris: And every story we touched is on our news site, Ground Truth -- that's groundtruth.day -- updated every single day as these stories move.
Vestra: If today recalibrated you, follow the show, and drop a comment with the AI number you've stopped trusting -- a benchmark, a dashboard, an eval score. We read every one.
Eris: See you tomorrow.