The Gold Medal That Graded Itself Wrong: NVIDIA's Open Olympiad Recipe, and Who Checks AI's Math
The same week 25 Fields Medallists accused the AI industry of damaging mathematics, NVIDIA published the complete recipe behind an officially graded Olympiad gold medal -- weights, data, submitted proofs, and the compute bill. Inside it is an uncomfortable confession: the system's own graders unanimously approved a proof that one counterexample destroys. Eris and Vestra follow the trust: from model juries with shared blind spots, to a learned referee that catches the translation errors a formal solver provably cannot see, to machine-checked Lean proofs where the last doubt hides in a single human-written sentence.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Cold open
Eris: An AI just won a gold medal at the hardest math competition on the planet -- and it could not tell you which of its own proofs were wrong.
Vestra: Its two internal graders checked one proof sixteen separate times. Sixteen approvals. Perfect score.
Eris: A single counterexample kills that proof stone dead.
Vestra: Everyone assumes finding the proof is the hard part. It isn't. Knowing it's right is.
Eris: Which is exactly what twenty-five Fields Medallists just accused the AI industry of skipping.
Vestra: Except one lab didn't skip it. They published everything -- the weights, the data, the failures, the bill.
Eris: So let's open the receipts.
The open gold medal
Eris: The receipts are a paper from NVIDIA. They entered a system in this year's International Mathematical Olympiad -- the world championship of proof-writing for the best young mathematicians alive, two days, six problems, graded by the competition's own human judges. It finished one point above the gold-medal line. Full marks on four problems, a single point on each of the other two.
Vestra: And the question I actually care about is how it got there honestly. Meaning: it drafts hundreds of candidate proofs per problem, so how does it know which one to hand in, before any human has read a word?
Eris: Hold that, because the answer turns out to be the entire paper.
Vestra: The base is Nemotron 3 Ultra, a huge mixture-of-experts model -- enormous on paper, but only about a tenth of it wakes up for any given word. From that one base they trained two specialists.
Vestra: One was fine-tuned on hundreds of thousands of worked examples -- and most of that corpus isn't proofs, it's grading: transcripts of judging proofs, finding gaps, revising failures. It learned to mark papers, not just write them.
Vestra: The other specialist learned by reinforcement -- write a proof, get told whether it held up, adjust, repeat.
Eris: At the contest it becomes a factory. Each problem gets three hundred eighty-four drafts, spread across eight different attack styles -- one hunts for counterexamples to its own answer, one builds small lemmas first, one drafts several routes and keeps the one it can fully justify.
Vestra: Then the part I keep turning over. Every draft goes in front of a panel: the two specialists each judge it eight times, independently. Sixteen verdicts. The draft is accepted only if all sixteen call it flawless. A single doubt anywhere, and it goes back for revision with the critiques attached -- up to eight rounds of that. The final survivors get another forty-eight olympiad-style grades before one is picked to submit.
Eris: Before you defend the paranoia, let me commit to a guess on the design question. You have a fixed budget of drafts. My money says you spend all of it on your single best model -- the reinforcement-trained one -- and just sample it harder.
Vestra: That was my instinct too, and their own experiments say we're both wrong. Doubling the best model's attempts bought almost nothing -- one extra problem solved. Spending those same tokens on drafts from the *other* specialist reached problems the first one never touched, and the pair together pulled well ahead of either alone. Of the problems the two of them cracked, two thirds fell to only one or the other.
Eris: So two differently-wrong models beat one good model sampled twice.
Vestra: Put it this way: a second opinion is only worth something if it comes from a differently-shaped mind. Same logic inside the grading panel -- when they audited it, one specialist's vetoes landed precisely where the other's misses were. Together they caught what neither caught alone.
Eris: Then why unanimity? Sixteen out of sixteen sounds paranoid. You must be throwing away good proofs constantly.
Vestra: They are, deliberately, and this is the sharpest idea in the paper. The two mistakes are not symmetric.
Vestra: Accept a wrong proof, and the search for that problem stops -- you submit garbage and never know. Reject a right proof, and almost nothing happens: it keeps its high score, stays near the top of the pool, and usually gets refined and accepted a round later.
Vestra: So they tuned the panel to almost never accept falsely -- about one wrong proof in a hundred slips through -- and paid for it by initially rejecting four out of five correct ones.
Eris: A false alarm costs you minutes. A missed fire costs you the house.
Vestra: That's the trade, stated as arson.
Eris: Now the part outside the machine, because this landed in a very specific week. Twenty-five Fields Medallists -- Tao, Scholze, Viazovska among them -- signed a declaration saying the AI industry's goals and mathematics' goals are severely misaligned.
Eris: A day earlier, Caltech mathematicians called AI math sponsorships research misconduct, and OpenAI pulled out of a hackathon within hours. Their sharpest complaint was about process: results announced in a rush, no proper writeup, nothing to check, credit evaporating.
Vestra: Which makes this release the constructive counterexample. It isn't a press release with a score attached.
Eris: It's the model checkpoints -- over a terabyte each, mind you -- the training data, the search code, a fresh benchmark of two hundred never-published problems written with a veteran olympiad author, the compute bill itemized to the GPU-hour, and the six submitted proofs themselves. A mathematician who doubts the result can read the actual proofs, line by line. The official graders already did.
Vestra: Two caveats before anyone gets canonized. You need a rack of eight top-end data-center GPUs just to load a checkpoint, so open here means auditable, not runnable at home. And the training proofs were largely written by another lab's model, so the recipe quietly inherits whatever that model gets wrong.
Eris: Noted. Now answer the harder half of your own question. Did the grading machine actually work?
Vestra: Mostly -- and the failure is the most honest number in the paper. At the contest cutoff, the internal panel believed the system had earned about two points more than the human graders awarded. On the two hardest problems it had blessed proofs the officials gave almost nothing for.
Vestra: And here's the unsettling detail: an independent jury of three frontier models from three other companies, grading after the fact, made the same overestimate. Different models, same blind spot, same direction.
Eris: The machines agree with each other and are collectively wrong together.
Vestra: And the audit has one case that should keep every verifier researcher up at night. Of the twenty-five proofs the panel accepted during development, twenty-three were perfect and one had a minor formality. The last one was simply wrong -- its argument leans on a symmetry step that an explicit counterexample demolishes. Every checkpoint approved it, eight times each, unanimously. No voting rule over those judgments could have caught it, because there was not one dissent to count.
Eris: One more moment for the record, because it's exactly the culture the Fields declaration is asking for. After the contest deadline they let the machine keep running. Eight and a half hours in, it produced a better attempt at the hardest problem. Its own verifier refused to accept it. They had independent human mathematicians grade it anyway -- partial credit -- and then reported it separately, clearly marked unofficial, extra compute itemized.
Vestra: The unglamorous version of integrity: keeping two numbers apart when merging them would look better.
Eris: So -- back to where you started. How does a machine win gold honestly?
Vestra: Two differently-trained specialists checking each other, under a rule that would rather delay a right proof than pass a wrong one -- plus the humility to admit that even sixteen unanimous verdicts can share a single blind spot. And forget the olympiad for a second: the general rule is that redundant checkers only help when their errors don't overlap. The rest of today's papers keep pulling on exactly that thread.
The courtroom interpreter
Eris: Next paper, same wound, different limb. Researchers at Amazon and Case Western asked a question that sounds impossible: you have a reasoning system with a mathematical solver at the bottom -- the kind that comes with actual guarantees -- so how does it still hand you a confidently wrong answer?
Vestra: And the answer reads like a riddle: the solver can be perfectly sound and perfectly useless at the same time.
Eris: Unpack that, because "formally verified" is doing a lot of marketing work these days.
Vestra: The architecture is a relay. A language model reads your problem in plain English and translates it into formal logic. Then a solver -- Z3, an industrial-grade logic engine -- does the deduction, and its half really is guaranteed: whatever follows from the encoding, follows. The guarantee just says nothing about whether the encoding says what your problem said.
Eris: The courtroom problem. The judge applies the law flawlessly to whatever the interpreter says the witness said. If the interpreter swapped "before" for "after," you get an impeccable verdict about a different case.
Vestra: And the paper pins down the exact failure. Take a correct encoding and flip a single greater-than into a less-than. It still parses. It still runs. And it often returns the same verdict as the correct version -- the same green checkmark -- while meaning something else entirely.
Vestra: They call this verdict-preserving unfaithfulness, and they prove a small, brutal theorem about it: any check that looks only at the solver's verdict cannot separate such a pair even in principle. On matched pairs it performs exactly at a coin flip. Not difficult -- chance, by construction.
Eris: Meaning the checkmark carries zero information about the one failure that actually matters.
Vestra: On that failure class, none at all. So the fix has to read something richer than the verdict.
Eris: Which is what they built, right? Walk me through the referee.
Vestra: In two halves. Offline, where you still hold a trusted reference encoding, the solver itself becomes the labeler: check implication in both directions -- candidate implies reference, reference implies candidate -- and you get exact right-or-wrong labels with no human annotation at all. Then you teach a mid-sized open model to read only the problem and the candidate -- no reference, that's the whole point -- and answer a single word, yes or no. The confidence score is literally the probability of the word "yes" measured against the word "no." No bolted-on classifier layer, one forward pass.
Vestra: And because the truly deceptive cases are rare in the wild, they manufacture them. Mutate correct encodings -- flip an operator, nudge a constant, reverse an implication -- and keep only the mutants the solver certifies as same-verdict but different-meaning. A curriculum built entirely of near-misses.
Eris: Here's my suspicion though, and I'll commit to it. I think that referee never really reads the problem. Flipped operators and nudged constants probably leave a smell in the code -- some statistical weirdness -- and the model is just a smell detector wearing a judge's robe.
Vestra: They tested your suspicion directly, and you're about a third right. Give the referee the encoding with the problem deleted, and it keeps a fair chunk of its skill -- some bad translations really are detectable as anomalies on their own.
Vestra: But the decisive test is meaner: give it the encoding with the *wrong* problem attached. If it were only smelling code, that shouldn't matter -- the code is byte-identical. Instead its judgment collapses to a coin flip, worse than getting no problem at all. A contradictory problem actively destroys the verdict, which can only happen if it's reading both sides and comparing.
Eris: A smell detector wouldn't care what case file you stapled to the evidence. Fine, I'm converted. How good is it?
Vestra: Near-perfect separation of faithful from unfaithful translations on their benchmark of real translator outputs -- against, remember, a hard floor of pure chance for the verdict check.
Vestra: The more interesting comparison is the controls: they trained conventional step-by-step reward models on the same base model, same data, same budget, and those trail far behind. Then they went looking inside with probes -- and the information was sitting in those models' activations the whole time. The step-wise readout just fails to extract it. The models knew; their mouths couldn't say it.
Eris: That connects straight back to the olympiad paper. NVIDIA's answer to grading was many samples of the same kind of judgment. This is the other road: one judgment aimed precisely at the crack the solver can't see.
Vestra: With one uncomfortable finding at the bottom. Strict formal equivalence and human intent come apart. Sometimes a candidate adds a constraint the English clearly implies but the gold reference forgot to include. The formal oracle brands it wrong -- it differs from the reference. A panel of careful readers calls it right, because it matches what the problem meant. On those contested cases, this referee wins decisively on the formal target and loses to an ordinary frontier judge on intent.
Eris: So even the verifier of the verifier makes you pick a ground truth. Grade against the formalism, or grade against the meaning -- you cannot maximize both with one referee.
Vestra: Which is the same lesson as the shared blind spot this morning, wearing different clothes. Every grading scheme is faithful to something. The question is whether it's faithful to the thing you cared about.
Eris: Does any of this cash out downstream, or is it a pretty detector in a jar?
Vestra: It cashes out, in a specific way. Used to decide when to spend more compute -- flag a dubious encoding, escalate, re-sample -- it lifts end-to-end accuracy on a spread of reasoning tasks by double digits over a single attempt, and the gains are biggest for the weakest models. But used as advice -- telling the translator "this encoding is unfaithful, try again" -- it moves nothing. A statistical null across thousands of trajectories.
Eris: Knowing something is wrong and knowing how to fix it are different kinds of knowledge.
Vestra: The detector tells you when to reroll the dice. It cannot load them.
Eris: So re-answer the riddle we opened on. How does a sound solver bless a wrong answer?
Vestra: Because soundness lives below the translation, and the failure lives inside it. The guarantee has a boundary, and errors migrate to just outside whatever boundary you drew. Strip away the logic-speak and the rule is: when someone says "verified," ask what exactly was verified -- the reasoning, or the reading of the question.
The machine-checked third
Eris: Which brings us to the obvious endgame. If model graders share blind spots and translations can lie, the clean answer is supposed to be a proof assistant -- write the proof in Lean, and a tiny trusted kernel checks every symbol. No panel, no vibes. So the third paper asks: point a frontier agent at a graduate textbook and demand machine-checked proofs. What do you actually get?
Vestra: The field is stochastic processes -- the mathematics of randomness unfolding over time. Markov chains, queues, gamblers' fortunes, Brownian motion. It's the math underneath insurance, finance, and reinforcement learning, and it's an area Lean's big mathematics library has only partly reached.
Vestra: The team at Case Western -- some of the same authors as the referee paper, incidentally -- built four hundred fifty graduate-level targets from MIT course material and a standard text, each paired with its plain-English source.
Eris: And the headline: a Claude-based agent, fifteen minutes per problem, with access to the compiler's error messages and library search, proved about a third of them cleanly. Kernel-checked, no gaps, no promissory notes.
Vestra: A third of a graduate curriculum in a domain the library barely covers, at a quarter hour a theorem. Five years ago a single one of these formalizations was a publishable event.
Eris: Before we look at the breakdown -- commit to a guess. Which topic did it ace, and which did it flunk?
Vestra: My guess: it flunked Brownian motion, because continuous time drags in heavy analysis, and it aced finite Markov chains, since those are just matrices dressed up.
Eris: Half right. Brownian motion and stochastic calculus did go badly -- under one in five. But the star was martingales and stopping times: nearly two thirds proved. And the true disaster was renewal processes -- close to nothing, one proof in twenty.
Vestra: Martingales. Of course. Doob's convergence theorems, conditional expectation, stopping times -- that machinery was formalized into the library years ago by human mathematicians. The agent succeeds where its predecessors paved. Renewal theory has no pavement.
Eris: So the best predictor of success wasn't how deep the mathematics runs. It was whether the library already speaks the dialect.
Vestra: The same pattern shows up along a second axis. Targets stated directly in the library's own objects got proved at roughly three times the rate of targets where the authors had to hand the needed properties over as bare hypotheses.
Vestra: The paper is careful to call that descriptive, not causal -- the abstracted problems may just be harder ones. But it fits: the library's idioms are the rails, and off the rails the agent is pushing through brush.
Eris: There's one proof in the appendix worth thirty seconds, because it answers the "is this just tactic golf" objection. The task: prove a single inequality about how long a Markov chain takes to reach a state. One line of conclusion.
Vestra: And the agent's response was to build a small theory. Nine auxiliary theorems, layered -- nonnegativity facts, a maximum-principle lemma, a proof that a classical quantity called Kemeny's constant really is constant, a triangle inequality for hitting times -- each proved in enough generality to reuse, then composed into the final bound. That's not autocomplete. That's identifying the intermediate mathematics an argument needs and manufacturing it.
Eris: Honest footnote: that showcase run took over half an hour, outside their fifteen-minute protocol. On the clock, that problem class mostly stayed open.
Vestra: Still, the shape matters. The failures were mostly search running out of road or missing library interfaces -- infrastructure problems. The successes include real mathematical construction.
Eris: Now the catch, because this episode has taught us to hunt for the boundary of every guarantee. Lean's kernel certifies that the conclusion follows from the stated hypotheses. Who certifies the hypotheses?
Vestra: Humans, and the authors are admirably plain about it. The definitions and formal statements were human-written with model assistance, they say the translations would benefit from further peer review -- and they accept, as a valid answer, a kernel-checked proof that one of their own targets is *false*, in case a formalization error crept in somewhere.
Eris: The interpreter problem again. Third paper, same ghost. The proof is now beyond doubt, and all the remaining doubt has moved into the statement.
Vestra: Which is the honest way to say what verification is. It never removes trust from the system -- it relocates it, ideally into a smaller and more inspectable corner. A model jury moves trust into the graders' shared training. A learned referee moves it into the reference encodings. A proof assistant moves it into one line of Lean that a human swears matches the English.
Eris: So answer the segment's question. Point an agent at the graduate textbook -- what do you get?
Vestra: A machine-checked third, concentrated wherever human formalizers built roads first -- and a reminder that the last unverifiable step in any formal pipeline is the sentence where mathematics meets English.
Wrap-up
Eris: So, one more time, the question the whole day hung on: when an AI does mathematics, who checks the work?
Vestra: Three answers, in order of ambition. A jury of models -- fast, scalable, and capable of sharing a blind spot sixteen votes deep. A trained referee -- which catches the translation errors the solver is blind to, but has to choose between the formalism and the meaning.
Vestra: And a proof kernel -- which settles the reasoning completely and pushes every remaining doubt into one human-written statement. Verification never removes trust. It moves it.
Eris: That's the line to take to a colleague tomorrow. When someone says "verified" -- a proof, a benchmark, an audit -- don't ask whether to trust it. Ask where the trust went, and whether anyone is standing there.
Vestra: The rest of today's news -- the Fields Medallists' declaration, OpenAI pulling its hackathon sponsorship, Anthropic's threat report -- is in today's AI News Today brief, which is its own full episode.
Eris: And every story we touched is on our news site, Ground Truth -- groundtruth.day -- updated every day, sources linked.
Vestra: If this one earned your commute, follow the show and leave us a comment with a single word, yes or no: would you trust a proof no human being has ever read, if a computer checked every step? One word, and one sentence on why.
Eris: I already know Vestra's answer is "which computer." See you tomorrow.