The Bug That Makes AI Models Score Better: Two Shipped Models Are Reading the Future
A five-line audit just caught two shipped AI models -- Zamba2 and NVIDIA's Nemotron-H -- silently reading future tokens, a defect the standard attention-mask check missed in every single one of 192 planted trials. Worse: the leak lowers loss and perplexity, so the bug improves the exact metrics that would have caught it. Eris and Vestra crack open the two-forward-pass detector, then follow the thread into agent land: a new benchmark showing nobody can find the step where a 145-step agent run broke, and Microsoft's AutoSaddler, which patches the agent harness automatically and survives its own unreliable diagnoses by making every fix earn its life on held-out tasks.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Cold open
Eris: There is a bug that makes an AI model score better. Which is exactly why nobody caught it.
Vestra: Every dashboard says the model is improving. Loss down, perplexity down, ship it.
Eris: Meanwhile the model is reading tokens from the future. Copying answers off a page it hasn't reached yet.
Vestra: And the check the whole field relies on -- look at the attention mask -- catches none of it. Zero.
Eris: Two of the affected models actually shipped. One of them carries NVIDIA's name.
Vestra: The detector that finds the leak runs in seconds. It's about five lines of code.
Eris: So let's do it -- how a model cheats at seeing the future, and why cheating made it look smarter.
The Mask Is Not the Model
Eris: Start with the question the whole thing hangs on. How does a language model even cheat at predicting the next word? The game is guessing what comes next. Where would a bug find an answer key?
Vestra: In the model's own input. During training and evaluation, the model sees the entire text at once -- past and future -- and one structural rule is supposed to keep each position blind to everything after it. The output at position forty may depend on positions one through forty and nothing else. That rule is what the word "causal" actually means. Break it, and position forty just reads position forty-one and hands it back as a prediction.
Eris: And the way the field verifies that rule is famously simple.
Vestra: You look at the attention mask. Attention is the operation that lets positions share information, and it comes with an explicit grid of allowed connections -- a triangle, past-only.
Vestra: And for years attention was the only place information could cross between positions, so inspecting the triangle really was inspecting the model. Correct mask, correct model, done.
Eris: But that stopped being true. The newer hybrid models interleave attention with state-space scans -- a running recurrence that sweeps along the sequence carrying a summary forward. A scan has no mask. There is no triangle to look at.
Vestra: Picture the mask as the seating chart that stops students copying off the kid behind them. The hybrid classroom quietly added a corridor where notes get passed hand to hand, and the proctor is still staring proudly at the seating chart. So a group of researchers in Seoul asked the obvious follow-up: what would a check that watches behavior instead of paperwork even look like?
Eris: And their answer is almost insultingly small.
Vestra: Two inputs, identical except for the very last token. Run both through the model with a recorder on every layer. Everything before that last position should come out identical -- and on a correct model it is identical, bit for bit, exact zeros. If any earlier position changed because you edited the future, that position saw the future. And the first layer where the difference appears is where the leak lives. Two forward passes. No training, no gradients, seconds on a laptop CPU.
Eris: Before you tell me how the showdown went -- my money says the mask check at least catches the crude bugs. The off-by-one stuff, the blatant shifts. It can't be useless.
Vestra: That was my instinct too, and we're both wrong. They planted faults deliberately -- eight flavors of leak, injected into eight different models at three depths each, nearly two hundred trials. Mask inspection caught none of them. Not most, none.
Vestra: And the two-pass audit caught every single one and named the exact layer it was planted in, every time.
Eris: How does the mask miss a fault you injected on purpose?
Vestra: Because every one of those faults lives in a layer's output, not in the mask. The mask attribute stays perfectly, correctly set while the tensor coming out of the layer carries future information. The inspector walks up, reads a valid seating chart, and goes home. On two of the tested models there wasn't even a chart to read -- pure state-space architectures have no mask attribute at all.
Eris: So that's the lab half. The reason this made our lead story is what happened when they pointed it at models people actually download.
Vestra: And I want to give them credit for the order they did it in. They did not audit blindly. They first read the source code -- the standard modeling library everyone loads these models through ships six implementations of the same chunked scan. Four match the reference implementation. Two of them, Zamba2 from Zyphra and Nemotron-H from NVIDIA, sum over the wrong axis in the chunk recurrence. Three lines, and the block is byte-for-byte identical in both files. Someone copied a bug from one model into the other.
Eris: So from reading code alone, they predicted those two leak and the other four don't.
Vestra: Then they ran the audit, and the prediction held exactly. Both leak, and each one starts leaking at precisely its own declared chunk size -- position two hundred fifty-six for Zamba2, one twenty-eight for Nemotron-H. Perturb one token at position four hundred, and everything before the boundary is untouched, exact zeros, then contamination switches on at the boundary and never stops.
Eris: Why the boundary exactly? That precision is what sells it for me.
Vestra: The scan processes the sequence in chunks and carries a state from each chunk into the next. The very first chunk's carry-in is hard-coded zeros, so the mis-oriented sum has nothing to corrupt there. From the first boundary on, the carry state can absorb contributions from later chunks -- the future flows backward into the past. And when they re-orient those lines to match the reference, the leak drops to exact zero on the same shipped checkpoint. One cause, proven by removal.
Eris: Okay, now the uncomfortable part, because the shrug is right there. A model peeks one chunk back. Nothing crashes. Who actually feels this?
Vestra: Anyone who trusted a number produced by these models. Peeking at the future makes next-token prediction easier, so the bug lowers training loss and lowers perplexity -- the exact metrics used to decide whether a run is working and whether a checkpoint is worth releasing.
Vestra: The failure improves its own report card. It is a smoke detector wired to switch itself off when it smells smoke.
Eris: Which is why nobody found it until someone wrote the five-line hook. Every normal signal was pointing the good direction.
Vestra: One scope caveat, because the authors are careful about it and we should be too. The defect sits in the plain PyTorch fallback path -- the code that runs when the optional fused GPU kernels aren't installed. That is most CPU work, most CI, and a large share of research reproductions. A production server with the fused kernels may take a different code path that nobody has audited yet, in either direction.
Eris: What I keep coming back to is that their own instrument lied to them twice, and they published both incidents instead of burying them.
Vestra: Those are my favorite pages of the paper. First, three checkpoints came back perfectly clean -- and it turned out the loaded models weren't responding to their input at all. Broken load, constant output, so of course nothing diverged. A clean verdict means nothing unless you also plant a fault in that same loaded model and watch the detector fire.
Eris: And the second one is worse, because it nearly cost them the whole finding. Their first sweep ran on sequences of forty-eight tokens -- shorter than one chunk. The buggy code path never executed. Zamba2 came back spotless.
Vestra: If the test sequence is shorter than the model's internal chunk or window, the machinery you're trying to test never runs in its general form. They only found the leak after questioning their own test length. So the audit needs two disciplines: prove the detector is live, and test longer than every internal size parameter. Otherwise you're certifying an empty room.
Eris: One more choice worth mentioning -- they released no code. On purpose. For a paper about a detector, that's a statement.
Vestra: Their argument is that the method is five lines on top of standard hooks, and an independent reimplementation is a stronger reproduction than running someone else's binary. They published the full logs and per-layer measurements instead. It will annoy people, and I think it's defensible. They also did the responsible thing upstream -- the Zamba2 fix is filed against the library, and the Nemotron-H instance is disclosed in the paper with the same two-line patch.
Eris: So close it out. Why did the broken models score better instead of worse?
Vestra: Because seeing the future makes prediction easier, and easier prediction is exactly what our metrics reward. The bug and the improvement are the same event. And forget the seating chart -- the general rule is: verify the behavior, never the declaration, and never accept a clean result from a detector you haven't watched fire.
Nobody Can Find the Step That Broke It
Eris: Segment one was a bug hiding in a model. This one is about bugs hiding in agent runs, and it opens on a question every agent developer has lived: an agent works for a hundred and forty-five steps, fails the task, and you have to find the one step where it actually went wrong. Can anyone do that?
Vestra: A new benchmark called LongRCA finally measures it, and the answer is brutal. But guess first. State of the art, full transcript in hand, how often does the best method point at the exact step?
Eris: I'll say half the time. The transcript is all there, it's reading comprehension.
Vestra: About one time in eight. The strongest existing method finds the exact root-cause step roughly once in eight failures. And the method the authors built specifically for this, which beats everything else on every measure, still only gets the exact step about one time in four.
Eris: That's the kind of number that says the problem is different from what it looks like. So walk through what they built, because the construction is the interesting part.
Vestra: Eleven hundred and forty genuinely failed agent runs -- and "genuinely" is the design choice that matters. Most earlier work injects errors: a researcher corrupts a step on purpose and asks whether a method can find it. That produces clean, findable mistakes. These are failures the agents produced on their own, across five domains -- software repair, terminal work, travel planning, service tool use, web browsing.
Vestra: Median of a hundred and forty-five steps per run, and human annotators marked two things for each: which role in the workflow was responsible, and the earliest step that introduced the decisive error.
Eris: Give people the flavor of one, because "decisive error" sounds abstract until you see it.
Vestra: There's a software-repair run in the paper that's perfect. At step thirty-seven, the diagnosis agent hands the execution agent a repair plan built on the wrong API. The executor follows the plan faithfully, competently, for over a hundred more steps, and reports the job complete at step one sixty-three. Then the test suite runs and every required test fails. The root cause is step thirty-seven. Everything after it is a flawless execution of a wrong idea.
Eris: Which is why this is hard. The decisive step doesn't look like a mistake. It looks like a perfectly reasonable plan, and the run doesn't start visibly failing until much later. The needle looks like hay.
Vestra: And in half these trajectories, more than fifty steps of recorded execution follow the root cause before the run ends. When methods miss, they don't miss by a little -- the average distance between the predicted step and the true one is around forty steps.
Eris: Now the part I flagged when I read it, and I want your honest read. The labels are human judgments. How much can you trust a human pointing at step thirty-seven in a hundred-and-forty-five-step log?
Vestra: The authors publish the uncomfortable number themselves: when two annotators labeled the same run, they agreed on the exact step less than half the time. They agreed on the responsible role about two-thirds of the time. Each annotation took a trained grad student half an hour or more. So part of that one-in-eight ceiling is genuine ambiguity -- if two careful humans split between step sixty-two and step seventy-one, a method answering seventy-one isn't exactly wrong.
Eris: That's not a flaw they hid, it's kind of the finding. Even the ground truth is fuzzy at the step level.
Vestra: Which shows up in the results as a clean split. Naming who broke the run and naming where it broke are different problems. Their method identifies the responsible role about half the time -- but the exact step only a quarter of the time. Blaming a component is a coarse call among a handful of candidates. Picking one step out of a hundred forty-five, when the step looked reasonable at the time, is a different sport entirely.
Eris: Think of a plane incident investigation. "The failure originated in maintenance" is one level of finding. "It was the torque check skipped on this date on this bolt" is the finding that changes anything. The field has been reporting the first and quietly implying it had the second.
Vestra: And their method is honest about how it claws out its lead. It doesn't read the whole log in one gulp. It cuts the trajectory into segments, summarizes each, pulls out candidate error steps, and then -- this is the clever bit -- traces each candidate backward to the handoff instruction that preceded it. If the instruction already contained the error, the instruction is the root, not the step that obeyed it. That one rule is basically the step-thirty-seven story turned into an algorithm.
Eris: So ask the opening question again. The agent fails at step one sixty-three -- can anyone find the step that broke it?
Vestra: Mostly no. Best case, one time in four exactly, and even humans agree with each other less than half the time. What we can do somewhat reliably is name the guilty component. Hold that thought, because the next paper is an automated improvement loop whose first stage is exactly this diagnosis step -- and now you know how shaky that stage is.
Patch the Harness, Not the Model
Eris: So here's the tension we just set up. Root-causing an agent failure is nearly unsolved -- one in four at best. And yet Microsoft just published a system that reads agent failures and automatically fixes the agent, with real gains. How can both be true?
Vestra: Because of what it fixes and how it decides to keep a fix. The system is called AutoSaddler, and the first thing to understand is that it never touches the model. It edits the harness -- the scaffolding around the model. System prompts, tool definitions, retry logic, the rules about when to stop. All the stuff that today gets tuned by hand, by expensive people, and doesn't transfer between projects.
Eris: And the harness genuinely moves scores. We've covered runs where swapping the scaffolding shifted results more than swapping the model did. So automating that tuning is going after real money.
Vestra: The loop reads like a training loop, except the thing being trained is code and text instead of weights. Run the agent on a small batch of tasks. Take the failures. Have a diagnosis agent actually dig through the execution traces and the harness source -- not just glance at them -- and write a structured patch: a new tool, a changed rule, a hook. Test the patched harness on the same batch. If it improved, test it again on held-out tasks it has never seen. Only patches that survive that second gate get kept.
Eris: And that second gate is the answer to my opening question, right? The loop doesn't need the diagnosis to be reliably correct.
Vestra: Exactly the point. A wrong diagnosis produces a patch that fails validation and dies. The diagnosis proposes; the held-out evaluation disposes. It's the same lesson as the mask paper wearing different clothes -- don't trust the explanation, test the behavior.
Eris: Say what it bought them. Without the raw scoreboard.
Vestra: Three unrelated agent tests -- a general assistant working a simulated phone, real software fixes on real repositories, and command-line tasks. One unchanged procedure lifted all three by about the same amount, roughly one extra task solved out of every ten attempted. On the terminal one it edged past a harness that human experts had hand-tuned. And it got there cheap -- its best result used about a tenth of the failure traces the closest automated rival burned through.
Eris: A single tuned number on one test is noise. The same lift on three unrelated ones is a pattern.
Vestra: The part I'd actually keep is the ablations, because each one is a correction to something people currently do. Version one: replace the deep dig with a single "reflect on what went wrong" call -- the standard move in prompt-optimization pipelines -- and the gains shrink. There's a lovely case study where the shallow reflection blames a wrong relative path, which sounds completely plausible, except the trace shows the agent never checked where the parent folder lived at all. It invented a tidy story instead of reading the evidence. The deep version found the real hole and patched the harness to resolve paths automatically.
Eris: Plausible-but-wrong diagnosis. Which is exactly the failure mode you'd predict from the last segment.
Vestra: Version two: let the patcher edit anything, unconstrained. It collapses into laziness -- nearly all of its patches become little text tweaks to prompts, because text is easy to write. Forced structure pushes it toward the patches that actually pay: new tools, control-flow changes.
Vestra: And those code-level patches fix about as often as the text tweaks while causing half as many regressions. A prompt edit sprays over everything the agent does; a scoped tool change touches what it touches.
Eris: The diva patches versus the plumbing patches. The plumbing wins.
Vestra: And version three is the biggest drop of all: remove the held-out validation and the reflection step, and performance falls below where you started. There's a beautiful disaster in the logs -- one variant patched a hook to forcibly redirect the agent's most-used messaging tool to a new tool it had just invented. Fixed the batch in front of it, broke unrelated scenarios everywhere. The full system caught an almost identical over-broad patch of its own and refused it.
Eris: That's the diary-versus-rule distinction. Everyone's building agent memory right now -- piles of past episodes for the agent to consult. This says the thing worth keeping isn't the episode, it's the validated correction extracted from it. A diary entry says "this happened once." A rule that survived held-out testing says "this generalizes."
Vestra: Now the skepticism, because this corner of the field has burned us before. Optimizing a harness against a benchmark's validation split is precisely where overfitting lives, and "it generalizes" is here judged by the same benchmark family it was tuned inside. They did split things carefully -- trained on some repositories and personas, tested on entirely different ones, so it isn't memorizing tasks. But we have covered systems that rewrote their own harness, gained big on the leaderboard, and flunked ordinary office work. Gains like this are worth exactly what the benchmark is worth.
Eris: The code being public is the fastest way to find out which kind this is. One more thing argues for real substance though -- the harness tuned with the big model still helped when they swapped in a much smaller, cheaper model underneath. A harness that only worked for one model would smell like memorized quirks. One that transfers looks like it fixed actual holes in the scaffolding.
Vestra: So, closing the loop. How does automated repair work when diagnosis is nearly unsolved?
Eris: Because it doesn't bet on the diagnosis. Every explanation has to buy its survival by improving held-out tasks, and wrong stories die at the gate. Diagnosis proposes, validation decides.
Wrap-up
Eris: So back to where we started. If a bug makes the score better, what in your pipeline ever catches it?
Vestra: Nothing you already have -- that's the answer. The metric can't police a failure that improves the metric. You need a check that's independent of the score, cheap enough to run every time, and proven live -- plant a fault and watch it fire before you believe a clean result.
Eris: That's the one thing to walk away with today. Green dashboards don't mean correct. Two forward passes on a CPU tell you whether your model can see the future, a held-out validation gate tells you whether your fix is real, and in both cases the trick is the same: test the behavior, not the story about the behavior.
Vestra: The rest of today's news -- the OpenAI chip numbers, the compute centralization argument, the anonymous model that half a million developers are feeding their code -- is in the AI News Today brief, which is its own episode right next to this one.
Eris: And every story we touched, plus the ones we didn't, is on our news site, Ground Truth -- that's groundtruth.day -- updated every day with the primary sources linked, so you can check us the way these auditors checked the models.
Vestra: If this one earned it, follow the show and leave a comment with the model you'd point the five-line audit at first. I have a list going and I want yours on it.
Eris: See you tomorrow.