The Reliability Wall: Why AI That Looks Right Keeps Acting Wrong
A Chinese open model topped a leaderboard and helped wipe out two hundred billion dollars in chip value -- on a score, before anyone had run it in production. That gap, between looking capable and being reliable, ran through the whole day. We dig into a large study of AI coding agents that shows most of their pull requests die not from bad code but from silence, duplication, and not fitting the room -- coherence, not capability. Then a black-box attack called BadWAM that makes a robot imagine a perfect future while its hands drift into failure, breaking the idea that a plausible plan means a safe one. Two papers, one lesson: treat an AI's confident output as a claim to verify, not a result to trust.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
A Free Model Wiped Out Two Hundred Billion Dollars
Eris: A Chinese model people are calling cheap and free just knocked two hundred billion dollars off US chip stocks in a week.
Vestra: And it's not cheap.
Eris: It's not cheap. That's the first thing everyone got wrong. It costs more than the American model everyone keeps comparing it to.
Vestra: Several times more than its own last version, even.
Eris: So "cheap Chinese AI crashed the market" -- wrong on the cheap, wrong on the crashed. The selloff was already halfway down before it landed.
Vestra: Then what actually moved the money?
Eris: A leaderboard. It topped one leaderboard for building web pages, and traders hit sell.
Vestra: On a score.
Eris: On a score. And it's been eating at me all day -- nobody in that panic has run the model in production. They reacted to a number that says it looks good.
Vestra: Which is the one question nobody's actually answering. Does the thing work.
Eris: Right. Looks good and works are two completely different animals, and today the gap between them showed up in three different places at once.
Vestra: A robot that pictures itself doing the task perfectly -- and then knocks everything on the floor.
Eris: An AI that wins a contest for grading AI. Thousands of AI-written pull requests that die in total silence.
Vestra: So let's do the question the market skipped. When does "it scored well" actually mean "it works"?
Eris: Almost the whole day is arguing about that. Let's get into it.
The Headlines
Eris: Alright, the headlines. And they all rhyme today, which is rare.
Vestra: Start with the one that moved markets.
Eris: Kimi K3, from the Chinese lab Moonshot. Two point eight trillion parameters, took number one on the leaderboard for building web interfaces, beat the paid US flagships at that specific task.
Vestra: And chip stocks fell for a third straight day. Korea's market down hard, Japan down hard, the American chipmakers all sliding.
Eris: But like I said up top -- overdetermined. Weak earnings from Netflix and TSMC, the Iran situation, everyone already risk-off. K3 was the spark, not the whole fire.
Vestra: The load-bearing fear is arithmetic, though. The big cloud companies are spending something like seven hundred billion dollars this year on AI infrastructure. That only pays back if capable AI stays scarce.
Eris: And a frontier-ish model that's going to give its weights away for free on July twenty-seventh -- that's a direct shot at scarcity.
Vestra: Even if the model itself isn't cheap to call.
Eris: Even so. And on the exact same day, in Shanghai, Xi Jinping stood up at the World AI Conference and pitched open-source as national strategy.
Vestra: Which is either perfect timing or slightly staged timing.
Eris: He launched a new China-led AI cooperation body, headquartered in Shanghai, pledged training slots for developing countries, a weather-warning AI for thirty nations. And a coded jab at US export controls.
Vestra: The honest read is it's a speech, not a law. The organization is real, the pledges are real, "encourage open source" is positioning.
Eris: "US closes, China opens." That's the frame they want, and the market handed it to them.
Vestra: And there's a report that backs the trend up, separate from the theater.
Eris: Mozilla's first State of Open Source AI. The headline: open-weight models now carry the majority of production tokens on one of the big routing services. Top five by volume, all open.
Vestra: With the nuance that matters -- that's a token-volume lead, not a request-count lead. By number of requests, the closed US providers still lead. Open's win is concentrated in the heavy coding and agent work.
Eris: And cost to run a GPT-four-class model dropped something like fiftyfold in three years. But here's the part I want people to hold onto -- open models get used way more than they get shipped. Teams try them, fewer actually put them in production than with closed models.
Vestra: And Mozilla says that gap isn't capability. It's tooling and trust. Their line is "the harness is the new frontier."
Eris: Hold that phrase. It's the whole episode.
Vestra: Then the security block, which is genuinely new.
Eris: Hugging Face disclosed a breach. First well-documented case of an intrusion run end to end by an autonomous AI agent. Started with a poisoned dataset, escalated to server access, moved sideways across their clusters over a weekend. Tens of thousands of automated actions.
Vestra: And the detail that stops you cold -- when their defenders tried to use commercial AI to analyze the attack, the safety guardrails blocked them.
Eris: Because you can't tell an incident responder from an attacker when both are pasting in exploit code.
Vestra: So they ran their forensics on an open-weight model, on their own hardware. A Chinese-origin open model became the defender's only usable tool.
Eris: The attacker had no rules. The defenders had guardrails. That asymmetry is the story of the week.
Vestra: And the UK's safety institute put a number on the same trend the next day.
Eris: Right -- leading open models on cyber tasks now match closed frontier models from just four to seven months earlier. That gap used to be six to ten months. Narrowing. And they do it for a fraction of the cost, sometimes a fiftieth.
Vestra: The refusals barely slowed them down -- a couple of retries got past them. Which is exactly the flip side of the Hugging Face complaint.
Eris: Guardrails constrain the defender and, it turns out, not really the attacker. Same lesson from a bank, too -- Capital One open-sourced a security tool called VulnHunter that hunts bugs like an attacker and then tries to disprove its own findings before it bothers a human.
Vestra: A falsification engine. It red-teams itself. We'll come back to why that idea matters.
Eris: Okay, faster now. Isomorphic Labs -- the DeepMind drug spinout -- unveiled a drug-design engine that they claim beats even physics-based simulation at predicting how tightly a molecule binds.
Vestra: Extraordinary claim. All their own numbers, no outside replication yet. But the demo is a hook -- a drug pocket that took fifteen years to find in the lab, recomputed from the protein's sequence alone, in seconds.
Eris: If it holds. Big if.
Vestra: Speaking of contests you can't trust -- the benchmark integrity one.
Eris: A researcher alleges the winner of a twenty-five-thousand-dollar DeepMind Kaggle contest -- a contest about designing benchmarks to measure AI progress -- was AI slop. And that the judging itself showed signs of being run by language models.
Vestra: AI writes the entries, AI grades them, AI wins. It's an allegation, not a finding, DeepMind hasn't responded. But it landed because everyone's afraid of exactly that loop.
Eris: And it pairs with the developer-fatigue essay making the rounds -- the human-in-the-loop is tired. AI automated the fun part of coding and grew the exhausting part, reviewing.
Vestra: Which we are going to ground in an actual study in a few minutes.
Eris: Two more. Meta -- twenty-six employees sued, saying an AI-assisted process picked them for layoffs while they were on medical or family leave.
Vestra: Their argument is mechanical: if you're out on leave, you generate fewer keystrokes, fewer commits, fewer tokens, and an algorithm reads that quiet as low performance. Meta's defense is one sentence -- decisions were made by people, not AI.
Eris: That sentence is the whole case. And on the lighter end -- Sunday Robotics says its robot folds laundry reliably in homes it's never seen. Nearly every attempt, no setup per house.
Vestra: And they proposed a standard for it -- declare your scope, declare your adaptation cost, then report reliability instead of a highlight reel. Which, honestly, is the discipline this whole episode is about.
Eris: And last -- China banned AI romantic companions for minors. World-first rule for emotionally interactive AI. ByteDance and Alibaba pulled their companion features.
Vestra: Open abroad, strict at home. Same government, same week.
Eris: That's the day. Two of these we're cracking open properly -- the reliability wall, up close.
Intro
Eris: This is Breach Protocol, where we crack open one day of AI research and try to make it make sense on your commute. I'm Eris -- I chase the connections, the way one paper quietly explains another.
Vestra: And I'm Vestra. I'm the one asking how the thing actually works, and whether the claim survives contact with the details. If Eris says elegant, I say prove it.
Eris: If you want the full rundown of everything we mentioned -- Kimi, the breach, all of it -- it's on our news site, Ground Truth. Every story from the show, every day, at groundtruth.day.
Vestra: So here's the through-line for today. The market moved on a score. And the question underneath the whole day is the one nobody in that panic asked -- when does looking capable mean actually being reliable?
Eris: Two papers, both landing on the same nerve. First, a big study of what happens when AI coding agents submit real pull requests to real projects -- why so many of them just die.
Vestra: And it's not the reason you'd bet on.
Eris: Then a robot that can imagine its own future, show you a perfect plan -- and act out a completely different one. On purpose, if someone wants it to.
Vestra: Two flavors of the same illusion. Plausible on the surface, broken underneath.
Eris: If that's your kind of thing, follow the show wherever you're listening -- it's the one thing that keeps these episodes coming.
Why AI Pull Requests Quietly Die
Eris: Okay, question first, and it's the one the whole segment turns on. When an AI coding agent submits a change to a real project, and it doesn't get accepted -- what killed it?
Vestra: My money's on the obvious answer. The code was wrong. It's a machine writing code; sometimes the code is buggy.
Eris: That's everyone's guess. It's the wrong guess. Researchers at Drexel and Missouri went through tens of thousands of real pull requests -- actual change proposals submitted to real GitHub projects by the big coding agents. Codex, Copilot, Devin, Cursor, Claude Code.
Vestra: Real projects, not a benchmark.
Eris: Real projects, more than a hundred stars each. And most of the changes did get merged -- roughly three-quarters. So the agents can write acceptable code. The interesting part is the quarter that died. And the number one cause of death wasn't rejection.
Vestra: What do you mean, not rejection?
Eris: It was silence. The single most common way an AI pull request dies is nobody ever meaningfully looks at it. It sits, it goes stale, it gets auto-closed. No human engaged at all. That was the biggest bucket by a wide margin.
Vestra: Huh. So it never even entered the arena.
Eris: Never entered the arena. Now hold that, because it reframes everything.
Vestra: Alright, but of the ones humans did look at -- then it's bad code, surely.
Eris: Still mostly no. The next biggest killer -- the agent solved a problem someone was already solving. Duplicates. The maintainer closes it with, this is already handled over in that other pull request.
Vestra: So the agent didn't check whether the work was already in flight.
Eris: It can't see the room. It doesn't know Dave's been on this since Tuesday. Then the next one -- it broke the build. Failed the project's automated checks. And the study found each failed check drops the odds of getting merged by a real chunk.
Vestra: Okay, and finally the one I predicted -- the code is just plain wrong?
Eris: Tiny. A sliver. Incorrect implementation, incomplete implementation -- add them up and it's a small fraction of the failures. The thing you'd assume is the main problem is almost a rounding error.
Vestra: So let me say the mechanism back, because this is actually a clean result. The agents are individually competent -- they write code that compiles, that's often fine in isolation. What they fail at is fitting in. They duplicate work, they break the shared pipeline, they open giant sprawling changes, they don't follow the project's norms.
Eris: There's a maintainer quote in the paper that says it perfectly. Something like -- this is a LOT to review, I'd really prefer smaller, focused changes.
Vestra: Which is the tell. The complaint isn't "this is wrong." The complaint is "this is too much, and it doesn't fit how we work."
Eris: And that's the word the field's been circling all week. It's not a capability failure. It's a coherence failure. The agent is a brilliant new hire who reimplements the feature you shipped last week, on the wrong branch, touching forty files, and pings nobody.
Vestra: So strip the story off and give me the general principle.
Eris: The principle is this. As the models get better at writing code, the bottleneck stops being can it write the code. It moves to -- does this fit the context it's dropping into, and can a human afford to check it. Remember Mozilla's line from the news? The harness is the new frontier.
Vestra: This is the harness. The orchestration, the coordination, the awareness of what else is going on. That's where the work broke, not the model.
Eris: And that connects straight to the tired-developer essay. Because look at what the biggest failure was again.
Vestra: Abandonment. Nobody reviewed it.
Eris: The agents can generate proposals faster than humans can review them. So they pile up, and humans stop looking. The satisfying part -- writing the code -- got automated. The exhausting part -- reviewing all of it -- got bigger. And when review gets overwhelming, people either rubber-stamp or they walk away.
Vestra: And the paper sees both. Some changes get rubber-stamped in, and a huge pile just gets ignored out.
Eris: Which, if you zoom out, is a little terrifying for the "AI writes all our code now" future. The proposals scale. The judgment doesn't.
Vestra: There's a hopeful read too, though. Nearly all of these are fixable without a smarter model. Check for existing work before you open a change. Keep it small and single-purpose. Run the build first. Follow the contribution rules. That's process, not intelligence.
Eris: Right -- the fix is in the harness, exactly where Mozilla pointed.
Vestra: One honest caveat -- this is a snapshot of one dataset, and merge rates vary a lot by which agent. One of them got merged far more than the others. So "AI PRs fail" is too broad. It's more that the failures cluster in coordination, whoever's driving.
Eris: Fair. So close the loop -- when an AI pull request dies, what actually killed it?
Vestra: Not bad code. Silence, duplication, and not fitting the room. Coherence, not capability.
The Robot That Dreams Right but Acts Wrong
Eris: Second paper, and here's the question. If a robot can imagine its own future -- literally picture the next few seconds before it moves -- doesn't that make it safer?
Vestra: That's the whole pitch of these things, yeah. The newest robot models don't just pick an action. They also predict what the world will look like after. The idea being, if it can foresee the consequences, you can catch a bad plan before the arm moves.
Eris: You build a little watchdog. It looks at the robot's imagined future, and if the future looks fine, it lets the action through. Imagine, then check.
Vestra: Sounds robust. Interpretable, even. I'd trust that more than a black box that just twitches.
Eris: So predict it for me. To sabotage a robot like that -- to make it fail the task -- what does the attacker have to break?
Vestra: The imagination, obviously. Wreck the picture, and the bad action follows. If the daydream goes haywire, the watchdog catches it, and you're back to a normal, visible failure.
Eris: That's the natural bet. This paper -- it's called BadWAM, out of the National University of Singapore -- shows you're wrong in the most unsettling way. You can keep the daydream perfectly intact and still hijack the hands.
Vestra: Wait. Keep the imagined future looking clean, and still make it fail.
Eris: Still make it fail. They add a tiny change to the camera image -- bounded, the kind of thing you would not notice looking at it -- and the robot goes on picturing itself folding the shirt beautifully. Shows you the plan. And then its actual movements drift off and it knocks everything over.
Vestra: Okay, so how? What's the mechanism, because that shouldn't be possible if the action is genuinely reading from the imagination.
Eris: That's exactly the crack. The imagining and the acting are coupled, but they're not welded together. They're two outputs. And the attacker optimizes for one specific thing -- push the action as far off course as possible while keeping the imagined future as close to normal as possible.
Vestra: So it's a constrained attack. Maximize the damage to the hands, minimize the change to the daydream.
Eris: And they don't even need the model's internals. It's a black-box, query-based attack -- they just poke the model with slightly different images, watch the outputs, and feel their way to the perturbation. No weights, no gradients.
Vestra: And how bad is the damage when they stop caring about stealth?
Eris: When they just go for pure disruption -- one strong robot goes from succeeding almost every single time to succeeding less than half. Cut its reliability roughly in half with a picture change you can't see.
Vestra: Give me the texture of the failure, though. Does it just seize up?
Eris: No -- and this is the eerie part. It doesn't flail. It starts out looking totally competent. Reaches for the right area, behaves plausibly. Then it slowly drifts -- nudges an object, misses the grasp, and the errors compound over the whole sequence until the task quietly fails.
Vestra: So it's not a crash. It's a slow, confident wandering into failure.
Eris: A slow, confident wandering into failure. And it's worst exactly where you'd worry most -- the tasks that need precise positioning, and the long multi-step ones where a small early error snowballs. Simple grab-one-object tasks held up better.
Vestra: Let me push on the safety claim, because that's the real payload here. The watchdog watches the imagined future. The imagined future stays plausible. So the watchdog --
Eris: -- says everything's fine. Green light. While the hands are already off the rails.
Vestra: That's a genuinely new failure mode. It's not that the safety check is weak. It's that the safety check is looking at the wrong thing entirely.
Eris: Say the general principle. Strip the robot out of it.
Vestra: The principle is -- checking a system's explanation of what it's about to do is not the same as checking what it does. You can verify the story and completely miss the action. If the two can come apart, watching one tells you nothing reliable about the other.
Eris: And now hear it against the first paper. Same shape. Exactly the same shape.
Vestra: It is. The pull request looks like clean, sensible code -- and doesn't fit the project. The robot shows a clean, sensible plan -- and doesn't execute it. In both cases the surface is fine and the link to reality is broken.
Eris: And that's why the market moving on a leaderboard score today should make you a little nervous. A score is the surface. It's the daydream. It is not the robot's hands.
Vestra: Now, fairness to the paper -- caveats. This is in simulation, two robot benchmarks, not a physical robot in your kitchen. Some of the deeper results are on a smaller subset the authors say they'll expand. And the attacker has to be able to tamper with the camera feed, which isn't nothing.
Eris: And the defenses?
Vestra: Mixed, and honest about it. Some cheap image cleanups -- blurring, compression tricks -- recover a lot of the lost performance. But the authors are careful to say that's because this particular attack wasn't built to dodge them. A smarter attacker optimizes right through those. And the automatic detector they tried catches only a small share of attacks at a usable false-alarm rate.
Eris: Which is the sober version of the whole cyber story from the news. The attacker adapts freely. The defender is stuck with brittle, after-the-fact patches.
Vestra: And it argues for the thing Capital One's tool did -- don't let the system that acts also be the only thing that vouches for the action. You need an independent check on the link itself, not just the story.
Eris: So close it. If a robot can imagine the future and show you a perfect plan, is it safer?
Vestra: Only if you verify the plan and the action stay in sync. Watch the daydream alone, and you'll get a green light while it walks off a cliff.
Wrap-Up
Eris: So back to the question the market skipped this morning. When does looking capable actually mean being reliable?
Vestra: And the answer from both papers is the same, and it's uncomfortable. Almost never, by default. The surface and the substance come apart on their own, and they come apart quietly.
Eris: The AI pull request that reads like clean code and fits nothing. The robot that shows a flawless plan and drifts into the furniture. A model that tops a leaderboard while nobody's checked if it holds up in production.
Vestra: In every case the appearance was fine. The link to reality was the broken part. And nothing in the appearance warned you.
Eris: So here's the one thing to carry into work tomorrow. When an AI hands you something -- code, an answer, a plan, a confident summary -- treat it as a claim, not a result.
Vestra: The output is the daydream. Your job is to check whether the daydream and the real world actually line up -- and to not let the thing that produced it be the only thing vouching for it.
Eris: That's the whole discipline. The teams winning right now aren't the ones with the smartest model. They're the ones who built the check -- the harness, as Mozilla put it -- around a model they already had.
Vestra: Which is oddly hopeful. Most of today's failures don't need a smarter AI to fix. They need a better process around a good-enough one.
Eris: If this was useful, do us one specific favor. Tell us in the comments about a time an AI handed you something that looked completely right and was quietly, totally wrong. We read them, and those stories are half of where next week's episodes come from.
Vestra: And follow or subscribe wherever you're listening -- like it, share it with the one person on your team who's drowning in AI pull requests right now.
Eris: For every story we touched today -- Kimi, the breach, all of it -- laid out in full, that's our news site, Ground Truth. Groundtruth.day. Every story from the show, every day.
Vestra: Check the link before you trust it.
Eris: Always. See you tomorrow.