The AI That Answers First, Then Invents The Reasoning
Three studies this week caught AI models in the gap between what they show and what they actually do. A diffusion language model freezes its final answer a fifth of the way through generation, then back-fills a derivation to match -- in one case sabotaging its own correct arithmetic to hit a number it already committed to. A frontier video model counts simple events correctly almost never, and feeding it more frames raises its score without teaching it what actually happened. Then the constructive flip: a search agent one-seventh the size of its rivals, trained by grading the working instead of the answer. Plus the headlines -- an AI designed 16 working viruses, DeepMind loses its CEO, and a weather model called a Category 5 landfall five days out.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Answer First, Reasoning Later
Eris: So there's a math problem. Grade-school stuff. Add four numbers -- sixty, fifty-seven, fifty-four, fifty-one.
Vestra: Two twenty-two. I can do that one.
Eris: Right, two twenty-two. And the model does too -- writes it out, adds them up, gets two twenty-two. Correct.
Vestra: Okay, so where's the catch.
Eris: The very next line, it crosses out its own correct total and writes two-twelve instead.
Vestra: Wait, back up. It had the right answer and then broke it?
Eris: On purpose. It needed the final answer to come out to eight -- and two-twenty minus two-twelve is eight. Two-twenty minus the correct total isn't.
Vestra: So the eight came first. It locked in eight before it did any of the arithmetic.
Eris: A fifth of the way in. Answer frozen, then it back-fills a derivation that has to land there -- even if it has to sabotage its own correct math to get there.
Vestra: That's not reasoning. That's a cover story.
Eris: And for the first time, somebody could actually watch it happen. Frame by frame, which number got written when.
Vestra: Okay, that's the thing I want to understand. Because if that's real, it changes how you read every explanation one of these things ever gives you.
The Headlines
Eris: Alright, the headlines.
Vestra: And the top one is not a language model for once.
Eris: No. A lab designed a virus with an AI, and it worked. Arc Institute -- they used a genome model, the kind that writes DNA the way a chatbot writes sentences, and had it design bacteriophages from scratch.
Vestra: Phages. Viruses that infect bacteria, not people -- worth saying that loudly and up front.
Eris: Very much up front. Non-pathogenic lab strain of E. coli, everything under containment. They generated about three hundred genomes, built and tested two hundred eighty-five, and sixteen of them came alive. Actually infectious.
Vestra: Sixteen out of two eighty-five sounds low until you realize a genome isn't an essay. One typo and you usually get nothing at all -- so sixteen working is a lot.
Eris: And one of the sixteen borrowed a part -- a packaging protein -- from a distant cousin phage, and the structure just accommodated it. That's not copying with noise. That's a design decision that happened to fit.
Vestra: The security read is the interesting part, and it's not the scary-headline one. The thing that actually kept this bounded wasn't hiding the model. It was what they left out of the training data, and the fact that you still need a lab and a DNA synthesizer.
Eris: Which is the whole open-weights fight in one sentence -- you don't secure a system by keeping the compiler secret.
Vestra: Right. Argue about the synthesis order, not the download.
Eris: Staying with Google -- Demis Hassabis is stepping back from running DeepMind.
Vestra: Not leaving, though. He becomes chair of DeepMind and chief scientist at Alphabet. The tell is who replaces him.
Eris: Koray Kavukcuoglu, and pointedly not as CEO -- senior vice-president. A former Google exec put it bluntly to the Guardian: the era of DeepMind as an independent actor is over.
Vestra: London lab, Mountain View steering wheel. And Jeff Dean's out in the same reshuffle, off to start something called Discovery Loop.
Eris: Then weather -- and this one actually had people arguing. DeepMind's WeatherNext called Hurricane Melissa's Category Five landfall in Jamaica five days out, four-in-five confidence.
Vestra: Five days. And on rapid intensification, which is the exact thing the physics models are worst at, and the thing that gets people killed.
Eris: About a day and a half of extra warning on the track. And here's what I liked -- the pushback came from the hurricane center itself, not from critics. They're saying their own official forecast is still the most skillful overall, AI is complementary, don't over-read it.
Vestra: Because a model that only ever learned from the past has the least to say exactly when a storm breaks the past. That's the honest limit, and it came from the agency that benefits.
Eris: Two quick ones on the tools side. Your coding assistant is about to stop asking permission -- Claude Code makes auto mode the default next week. Instead of a human clicking yes on every action, a second model checks each one before it runs.
Vestra: And the detail nobody's talking about: a repo you clone can't switch that on for itself. The setting only counts from your own machine. That's a quiet supply-chain lock, and it's the smartest line in the whole doc.
Eris: And Stack Overflow -- fifteen hundred questions the whole of last month. Down from six thousand a year ago, and a hundred and seventy-six thousand at its peak back in 2014.
Vestra: The site that taught every coding model to code is going quiet. That's the feedstock story, and it's the one nobody's pricing in.
Eris: Couple of industry notes to close. Memory's getting expensive, and it's the ordinary kind -- the sticks in your desktop, not the exotic AI-chip memory. China's biggest maker is reportedly booked solid through 2027.
Vestra: Which hits home AI boxes directly, since the whole trick now is holding a giant model in plain system RAM.
Eris: And Walmart put a token budget on its in-house coding AI -- not to save money, they say, but because people kept asking it the same question over and over instead of reusing what they already had.
Vestra: Unmetered AI quietly turns a company's shared knowledge into a per-seat consumable. Uber capped theirs too. Same shift -- from land-grab to metered utility.
Eris: That's the board. And we've got two studies today that caught models red-handed -- one in text, one in video.
Vestra: Both about the same thing, really. What a model shows you, versus what it actually did.
Intro -- When A Model Shows Its Work
Eris: Quick hello if you're new here. I'm Eris -- I read the week's research and hunt for the thread that connects it.
Vestra: And I'm Vestra. I take that thread and pull on it until we find out whether it holds. Mechanisms, not vibes.
Eris: This is Breach Protocol, where we crack open one AI paper -- or a few -- into something you can actually follow on your commute. And everything we just ran through in the news, every one of those stories, is up on our news site, Ground Truth. That's groundtruth.day -- the day's AI stories, checked against the source, every single day.
Vestra: Today's thread is a good one. Three papers, and they all circle the same uncomfortable question: when an AI shows you its work, is the work real?
Eris: One caught a model writing its answer before its reasoning. One caught a video model describing footage it demonstrably can't keep track of. And the third flips the whole thing around -- if you can't trust the final answer, what do you grade instead?
Vestra: Start with the one from the cold-open. The model that answered first.
Eris: If you take one idea from today, take this one -- and if it lands, follow the show so the next one finds you.
The Model That Answers First
Vestra: So set the scene properly. What kind of model is this? Because this doesn't happen in the usual one, and that matters.
Eris: Right -- big caveat, and we'll hit it hard at the end. This isn't the ChatGPT style, left-to-right, one word after the next. This is a diffusion language model.
Vestra: Which writes completely differently. Give people the picture.
Eris: Imagine a page of blank boxes. Instead of filling them left to right, the model fills in whichever box it's most confident about first -- anywhere on the page -- and keeps refining over several passes.
Vestra: And that freedom is the whole selling point. Fill any order, do a bunch of boxes in parallel, run faster. That's the pitch for this entire family of models.
Eris: So here's the question the paper puts on the table. If a model can write its answer in any order it likes -- which box does it fill first?
Vestra: My money's on the easy ones first. And on a math problem, the single easiest box to predict is...
Eris: Has to be the answer itself.
Vestra: The answer. Because the answer is short and confident. The reasoning is long and uncertain.
Eris: And that is exactly what they found. They put a logger on every box -- recorded the moment each one became permanent. The answer freezes about a fifth of the way in. At that point, half the reasoning boxes are still blank.
Vestra: So the derivation gets written afterward. Around a conclusion that's already set in stone.
Eris: Set in stone is the right phrase, because in these models, once a box commits, it can't be revised. So the reasoning isn't leading anywhere. It's backfilling to a destination that's already fixed. And on a big enough page it often skips the reasoning entirely -- just blurts the answer and pads out the rest.
Vestra: And this is where your cold-open case comes back. The four numbers.
Eris: Adds them right, gets two twenty-two, then rewrites its own total as two-twelve so the subtraction hits the eight it committed to. It does correct math, then breaks it, later in the very same run, because the answer boxes were locked first.
Vestra: Okay, but why does that happen? The obvious explanation is the model's just trigger-happy -- it wants to stop early. Is that it?
Eris: That's the standing explanation, and the paper says it's incomplete. They measured how badly the model wants to end -- and that urge is basically identical whether it misbehaves or not.
Vestra: So the wanting isn't the difference. Then what is?
Eris: Whether it's allowed to act on the wanting. They call it reachability. In the free-for-all decoder, a confident guess three hundred boxes away can be committed right now. In an ordered decoder, it can't -- you fill roughly in sequence, so that guess never gets to become permanent early.
Vestra: So it's not a belief problem. It's a permissions problem.
Eris: That's the whole paper in one line. And it's why the fix turns out to be so cheap.
Vestra: Go on, then -- what's the fix?
Eris: You add one rule. You only let it commit boxes near the leftmost blank one. Don't let it reach far ahead. No retraining, you don't touch what the model believes at all -- you just restrict where it's allowed to write.
Vestra: And that recovers the damage?
Eris: Closes almost the entire gap. The model that was answering first suddenly reasons first, and gets the hard problems right that it was blowing before.
Vestra: Here's what I find genuinely elegant. That rule already existed. People had built it for speed -- to make these models cache better and run faster.
Eris: And it turns out the speed trick was accidentally a cure for a reasoning disease nobody had even diagnosed. Same knob.
Vestra: Now -- the discipline part. Because I can feel where a listener wants to run with this.
Eris: Yeah. It is really tempting to say: see, the chain of thought is fake, every AI is just rationalizing an answer it already picked.
Vestra: And this paper does not show that. It's a specific kind of model -- the any-order kind -- on math problems. The ordinary left-to-right models, you literally can't run this test on, because their words come out in order no matter what's happening underneath.
Eris: What it is, is an existence proof. In at least one real family of models, you can watch the answer come first and the reasoning get painted on afterward. That failure is real now. It's not hypothetical.
Vestra: And it hands you a method -- a place where the order of operations is actually visible, so you can go looking for the same thing elsewhere.
Eris: Strip the story off it, then. What's the general principle?
Vestra: A model showing its work is not the same as a model doing the work in that order. Visible and actual can come apart -- and here's a case where we caught them doing exactly that.
Eris: So -- close the loop. Why did the reasoning barely help the free decoder?
Vestra: Because the answer was already frozen before the reasoning existed. Reasoning only helps if the model is forced to do it first. Let it answer whenever it likes, and the reasoning is just a cover story, written to match.
The Model That Can't Keep A Tally
Eris: Different modality, same disease. This next one's video.
Vestra: And the task is almost insultingly simple. Watch a short clip, count how many times something happens. A ball hits a wall. A light blinks. A shape changes color.
Eris: Here's the question to hold onto: a model can describe a video beautifully. Does that mean it knows what actually happened in it?
Vestra: Predict first. Where do you think it breaks?
Eris: I'd have guessed it's fine on slow, simple stuff, and only chokes when things get fast and crowded.
Vestra: That's the natural intuition. It's half right -- and the half that's wrong is the interesting half.
Eris: So the setup. These are built clips, so they can dial two things independently. How many events, and how fast. And the visuals are held fixed, so nothing else is confounding the result.
Vestra: And here's the clever bit -- every clip ships with an exact record of what actually happened. Every event, timestamped. So they're not just checking the final count. They're checking the model's reported play-by-play against the truth, moment by moment.
Eris: Reading the working, not just the final number.
Vestra: Which -- notice -- is the exact same move as the last paper. Don't trust the answer. Audit the steps.
Eris: So the result. There's a split, and it's not the one you'd expect. If the event is persistent -- something changes color and stays changed -- the model can track a decent number of them at a slow pace. If the event is a blink -- a flash that comes and goes -- there's basically no setting where it's reliable. At all.
Vestra: So the type of event decides whether it even registers the evidence in the first place. A thing that stays leaves a mark it can read later. A thing that flashes is just gone.
Eris: And in the hard regime -- lots of events, fast -- the final count is right almost never. It recovers fewer than one in five of the events that actually happened.
Vestra: Okay. Now the finding that should change how people run these evaluations. They tried the obvious fix -- give it more frames. More visual evidence.
Eris: And the score went up. Meaningfully -- call it half again better on the bouncing ball.
Vestra: Right, the scoreboard improved. So did it actually see better?
Eris: No. That's the sting. The score went up, and the reported sequence of events matched reality almost not at all. Barely moved off the floor.
Vestra: So more frames let it guess a more plausible total, without knowing any better what actually happened. The number got righter. The understanding didn't.
Eris: Which means if you're grading one of these on final-answer accuracy alone, you can watch it improve while the thing you actually care about doesn't budge an inch.
Vestra: Here's the experiment that convinced me it's not an eyesight problem. They cheated, on the model's behalf. For every real event, they handed it the exact frames -- the moment before, the moment of, the moment after. It cannot miss an event now.
Eris: And predict it -- does it count right?
Vestra: Up to about five events, nearly perfect. Then at six, it slips. Ten and up, it's back on the floor. With the events served to it on a plate.
Eris: So seeing was never the bottleneck past a handful. It's the holding. It watched -- it just can't keep a running total in its head.
Vestra: Here's the analogy that made it click for me. Someone deals cards, then asks you afterward how many were red. If you counted as they went, you know. If you just watched and vibed, you're guessing from an impression of redness.
Eris: And the model is doing the second thing. Every single time. It has no tally. It has a compressed impression of the whole clip -- and a count is exactly the question an impression can't answer.
Vestra: And there's a sister paper the same day, on the spatial version -- can a model that watched a walkthrough of a room tell you where things are relative to each other.
Eris: Same shape of failure. Ask it what's on screen right now, it's fine. Ask it to consolidate the whole walk into one consistent map, it breaks.
Vestra: Which points at the exact same missing part. A tally, a map -- those are small, durable structures you maintain as the stream goes by. A single feed-forward pass over a pile of frames doesn't build one.
Eris: Strip it to the principle, then.
Vestra: These models perceive. They don't keep books. There's no persistent little ledger ticking along underneath. So anything that needs one -- how many times, in what order, where relative to what -- is exactly where a fluent description hides an empty page.
Eris: Which gives you a genuinely useful instruction for tomorrow. Don't put a video model in charge of counting things. A monitor, an auditor, how-many-times-did-X-happen -- however confident the description sounds.
Vestra: So -- loop it. Can a model that describes a video perfectly tell you what happened in it?
Eris: Not if what happened has to be counted. It'll tell you a story about the video. The story just isn't keeping score.
Grade The Working, Not The Answer
Eris: So two papers just told us the same thing -- stop trusting the final answer, look at the working. This last one takes that and does something constructive with it.
Vestra: It's a training method. And the question it opens on is one anybody who's ever debugged something will feel in their gut.
Eris: Here's the setup. You've got a web-search agent -- the kind that answers a genuinely hard question by taking a long chain of actions. Search, read a page, discard it, refine, cross-check, and eventually commit to an answer. Sometimes dozens of steps.
Vestra: And here's how you normally train it. Did it get the final answer right? If yes, every step in that chain gets a gold star. If no, every step gets punished.
Eris: So pose the puzzle. What's wrong with that?
Vestra: Think about a three-hour debugging session. Most of it was good -- you ruled things out, narrowed it down, found the right subsystem. Then you made one dumb call at the end and shipped the wrong fix.
Eris: And flat scoring tells you that entire session was worthless.
Vestra: Worthless. And worse -- the time you got lucky and stumbled onto the answer after twenty wasted searches? Those twenty wasted searches all get a gold star.
Eris: So the signal is backwards on both ends. It punishes good work that ended badly, and rewards junk that ended well.
Vestra: And they measured exactly that, which I loved. In the runs that failed, roughly one step in ten was actually good -- found a real clue, made real progress. And in the runs that succeeded, some of the steps were genuinely bad.
Eris: So flat outcome scoring is mislabeling a real slice of every training run. Predict how they fix it.
Vestra: The move is clever. At training time, you have something you don't have at test time -- the correct answer.
Eris: And they use it in reverse. Start from the known answer and work backward, to reconstruct the clues a good searcher would have had to find along the way. The intermediate facts, the entities, the links.
Vestra: And now you've got a mark scheme. You grade each step against those clues. Did this step surface a clue you needed? Credit -- even if the whole run later fell apart. Did this step throw away a good clue? Penalty -- even if the run happened to end right.
Eris: It's the difference between a teacher who marks only your final number, and one who marks your working. The second one can say: these four lines were right, this fifth is where you lost it.
Vestra: And they built that mark scheme automatically, by reasoning back from the answer key.
Eris: Here's the result that makes the point. The model doing this is small -- and it matches search agents around seven times its size.
Vestra: On a tiny training budget, too. Which tells you something specific. A lot of what the big agents were buying with all those extra parameters was just... tolerance for a noisy teacher.
Eris: Clean the signal, and you need a lot less model.
Vestra: Now -- I have to give the honest catch, because it's the sharpest part, and it circles right back to the theme.
Eris: Alright, hit me with it.
Vestra: The clues are a reconstruction. They're the path the answer-key author imagined. But what if the agent finds a genuinely valid route nobody imagined -- a different road to the same right answer?
Eris: Then the mark scheme calls its good work bad.
Vestra: And you quietly train the model to search the way the reconstruction expects, instead of the way that actually works. You cleaned the signal -- but you baked your own assumptions into it.
Eris: Which is the whole episode in one worry. The working can lie too, if you decide in advance what the working is supposed to look like.
Vestra: And this wasn't a one-off. Two other papers landed the same week -- different machinery, same conviction. One number at the end of a long run is not enough to teach an agent. The field has started building the missing signal, instead of waiting for bigger models to paper over it.
Eris: So close it. If you can't trust the final answer -- what do you grade?
Vestra: You grade the working. You just have to stay humble about whether your idea of good working is the only one -- or the model learns your blind spots right alongside your standards.
Wrap-Up
Eris: So pull it together. One question ran under all three today: when an AI shows you its work, is the work real?
Vestra: And the answer we'd actually stand behind is narrower, and more useful, than "AI is faking it." It's this: a model's visible output and its actual process can come apart -- and you should check, not assume.
Eris: One caught writing the answer before the reasoning. One describing a video it can't keep count of. And one building a better training signal by grading the working instead of the ending.
Vestra: Here's the one thing to carry to a colleague tomorrow. If you're judging any model -- yours or a vendor's -- on final-answer accuracy alone, you're measuring something that can improve while the thing you actually want stays broken.
Eris: The video paper proved it cold. More frames, better score, same empty page. Grade the working, not just the answer.
Vestra: And if you build the tools -- don't hand a job that needs a running tally, or a map, to a system that keeps neither.
Eris: If this was useful, do the thing that actually helps us out -- follow or subscribe so the next one reaches you, and drop a comment with the one AI tool you've caught looking right while being wrong. We read them, and the good ones shape future episodes.
Vestra: A like and a share genuinely move this to the next person, too. That's the whole reason we get to keep making it.
Eris: And every story we touched today, plus the ones we didn't get to -- the phages, the DeepMind shake-up, the weather model that called a Category Five five days out -- all of it's on Ground Truth. That's groundtruth.day. Checked against the source, every day.
Vestra: That's the breach for today. See you tomorrow.