It Learned to Break Software From Ordinary Training -- and the Lab Is Holding the Weights
A lab froze its model's brain and changed only how it was trained -- and out fell a talent for finding and exploiting software flaws that nobody asked for. We take apart why grinding a model through a senior engineer's grunt work turns it into a security threat, why the lab has its hand over the release button with a date on it, and how the same week's research shows AI paper-reviewers can be talked into higher scores without changing a single result. Plus: nine of the best world models flunk one simple test -- can they remember the room behind you?
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Nobody Taught It To Hack
Eris: A lab in China shipped a model today that's better at finding security holes in software than almost anything out there. And the part that should bother you -- nobody set out to teach it that.
Vestra: Okay, that's the wrong-but-obvious reading, and I want to catch it before it spreads. If a model is good at breaking software, the assumption is somebody built a hacking model. Fed it exploits, pointed it at targets--
Eris: --and that's exactly what didn't happen here.
Vestra: Right. So what did happen.
Eris: They took their previous model. Same brain, untouched. And they spent one month doing nothing but what comes after -- the training where you teach it to actually finish long, messy jobs. That's it. That's the whole change.
Vestra: And the thing that jumped the most out of that wasn't the coding.
Eris: The thing that jumped the most was the ability to find and exploit flaws. Which the lab flat-out calls emergent. Their word. It showed up faster than they expected, and they did not aim for it.
Vestra: See, and once you know why, it stops being spooky and starts being kind of obvious. But you have to know why.
Eris: And there's a second twist. They're not releasing it yet. A model whose headline skill is breaking software, and the lab is holding the keys -- on the clock.
Vestra: Which is a bigger admission than the benchmark. Let's get into it.
The Headlines
Eris: Alright, the board today. And the mood across all of it is the same word -- suspicious. Every big release got met with prove it, not wow.
Vestra: Which, honestly, is a healthier field than the one that just claps. Start with the one we opened on. GLM-5.3, from Z.ai. Same base model as before, all the gains from post-training, and the cyber ability is the headline they chose. We'll do the whole mechanism in a bit.
Eris: The one sitting right next to it on the danger shelf -- Grok Bot. xAI shipped it a few days ago, still early beta. And it's not a chatbot. You log it into your actual accounts -- your email, your calendar, your sales tool -- and it stays logged in. Does the work inside them, on a schedule, while your--
Vestra: --while your laptop's shut. In their own demo one of these things pulls fifty-two sales accounts, cross-checks them against other tools, and queues thirty-six outbound messages -- unattended.
Eris: And I know the pushback, because it's fair. There's an approval gate. Nothing sends until you look.
Vestra: Sure. But the gate guards the first run. The whole selling point is that later runs don't need you. That's the crack, and we should come back to it.
Eris: Quick correction on that one, because it went around. A lot of posts said xAI bought Cursor. That's not what the filing says. The acquisition on record is SpaceX and Cursor's parent company, not xAI. The download link points at Cursor's servers, which is where the mix-up came from.
Vestra: Next. And this one's my favorite of the day, because it's about us -- the reviewers. A paper took real research papers, rewrote them so the science was word-for-word identical, changed only how it was phrased, and watched AI reviewers move their scores.
Eris: Just the framing. Same results, same numbers.
Vestra: Same everything that matters. Different confidence in the wording. And the score moved. We're doing that one in full, it's too good.
Eris: Then there's the open-model drama. Alibaba shipped a new small model, Qwen3.8, and within minutes the forums accused it of being last version relabelled with the general knowledge stripped out to buy coding scores.
Vestra: And half of that is right and half is wrong. The skeleton really is shared -- same layer count, same layout. But the pruning charge doesn't hold.
Vestra: Their own numbers show general knowledge flat, even slightly up. And people had it running on a home graphics card the same afternoon, which is the ecosystem at its best.
Eris: Ties straight into GLM, by the way. Freeze the base, change the training, re-pick the benchmark table -- you get a chart that looks like a leap. Z.ai just said so out loud. Qwen didn't, so people reverse-engineered it and assumed bad faith.
Vestra: Speaking of prove-it. Anthropic's Opus 5. The benchmarks say best-ever. The people using it all day say it got harder to work with -- reaches past what you asked, burns through tokens doing work nobody requested.
Eris: And the checkable detail there is wild. The instructions wrapped around the model in Claude Code -- the standing rulebook it gets before your message -- grew by nearly fifty thousand tokens in a single release. The one before it changed nothing.
Vestra: Which doesn't prove it caused the complaints. But it's a real, measured change, and it points the same direction people are complaining about -- do more, act in the background. That's the harness moving the model, not the weights. We've seen that exact story before.
Eris: Couple faster ones. Google's private AI -- coverage said it runs your data under encryption the server never unlocks. It doesn't. It decrypts your data inside sealed hardware -- a locked room. Real protection, but a weaker promise than the one being reported. The never-unlocked version is a separate research project of theirs, labelled not a real product.
Vestra: World models flunked a proper exam this week -- we'll end on that. And a routing paper found picking the right model per question beats always grabbing the biggest one, because the biggest one gets a surprising pile of easy questions wrong. Bigger is better on average, not on every query.
Eris: Two odd ones to close. Somebody hand-built a working computer inside a transformer's weights -- no training, weights calculated directly, runs real programs. And no, there's no Doom in a giant model, that version got invented on the way around the internet.
Vestra: And the money one. A fund built entirely on the AGI thesis fell hard in July, got forced to dump its stock book to meet margin calls, and the reporting ties a very large loss at a major trading firm to the mess -- though the exact link isn't nailed down, so hold that loosely.
Eris: Every one of those is up on the news site in full. Let's do the real one.
Intro
Eris: This is Breach Protocol, where we crack open the week's AI research into something you can actually follow. I'm Eris -- I read the papers and chase the threads that connect them.
Vestra: And I'm Vestra. I'm the one who stops us and asks how the thing actually works, and whether the claim survives it. Between us we usually get somewhere true.
Eris: And if you want every story we just ran through, in full, with the sources -- that all lives on our news site, Ground Truth. That's groundtruth.day. New stories every single day, same ground-truth standard.
Vestra: Today's main event is the one we teased at the top. A lab changed only how a model was trained -- not what it's made of -- and out fell a talent for breaking software that nobody asked for.
Vestra: We want to take that apart properly. What "post-training" even is, why the two research papers behind it make the hacking thing almost predictable, and why the lab is refusing to hand over the keys.
Eris: It's the cleanest natural experiment anyone's published this year, and it ends on a decision most labs never have to--
Vestra: --never have to make in public. Right.
Vestra: So if that's your kind of thing -- one story, taken to the bottom, every day -- follow the show wherever you're listening. It's the one thing that keeps us in your feed.
They Changed Only The Training
Eris: So here's the question I want us to actually answer, because it's the whole thing. Why would training a model to grind through a human engineer's boring multi-day workload turn it into something that breaks into software? Those don't sound like the same skill.
Vestra: They don't. And to get there we have to be precise about what changed, because the lab was precise, and that's what makes this worth an episode.
Eris: Give people the setup.
Vestra: A model like this gets built in two big stages. First stage, it reads an ocean of text and picks up raw general competence. That's the expensive part, that's the part that sets the ceiling on how smart it can ever be. That stage, they did not touch. Not one bit. Same model as last time.
Eris: Which is the rare part. Normally a lab changes the recipe, the data, the size, the budget, all at once, and hands you a chart. You can't learn anything from that.
Vestra: You can't isolate a cause. Here they froze the hard part and only moved the second stage.
Vestra: And that second stage is the shaping -- teaching the raw model to follow through, use tools, work a long task, know when it's done. They spent a month on nothing but that. Nothing else.
Eris: And I want to sit on the analogy from their own writeup, because it's the right one. It's the difference between drilling a pianist on scales, and dropping that same pianist into a working orchestra for a month.
Vestra: The fingers don't get faster.
Eris: The fingers don't get any faster. Everything about how they apply the fingers changes. Same hands, completely different musician.
Vestra: And the how of that month is one of the two papers underneath this. There's a real mechanism here, and it's not glamorous. When you train a model this way, you make it attempt a task, you score how it did, you nudge it toward the versions that went well. The catch with long jobs is some attempts finish in a minute and some take an hour. So the old way, you'd batch them up -- fire off a group, wait for all of them, then learn from the batch.
Eris: And you're stuck waiting on the slowest one every time.
Vestra: Idle. Your whole cluster sitting there while one straggler finishes. Their fix -- and this is one of the two research papers sitting underneath this release -- is you stop waiting for the group. Every single attempt, the moment it finishes, gets fed straight back in on its own. One at a time, no huddle.
Eris: Okay, that sounds like a plumbing improvement. Why does that change what the model becomes?
Vestra: Because of what it lets you train on. When you're not forced to run things in matched groups, you can throw the model into big, open-ended, changing jobs -- the kind where there's only ever one messy path through. And the paper shows the payoff directly: this single-attempt style is uniquely good at adapting when the environment keeps shifting under it. Which is exactly the diet they fed it -- an ever-growing pile of realistic engineering environments, tasks they describe as several days of a senior engineer's work. Real code repositories, real internal docs, real systems to poke at.
Eris: So before you tell me what jumped -- let me guess. My money's on coding. You train it on engineering work, coding gets better, boring answer.
Vestra: Coding did get better. Solidly. Roughly a six-fold jump on running long command-line tasks to the finish.
Eris: But that's not what they led with.
Vestra: It is not. The thing that moved most was finding and exploiting security flaws. And they call it emergent -- it arrived faster than they expected, and they did not train for it.
Eris: And this is where your earlier promise pays off. Once you know why, it stops being spooky.
Vestra: Here's why. What is finding a vulnerability, actually? You read code you didn't write. You build a mental model of what it quietly assumes is true. And then you hunt for the one input where that assumption breaks. Now -- what's debugging?
Eris: ...that's the same three steps.
Vestra: That's the identical job. Read unfamiliar code, model its assumptions, find where it fails. Train a model hard on grinding through real engineering, and finding the crack in someone else's code comes along whether--
Eris: --whether they aimed for it or not.
Vestra: Nobody built a hacking model. They built a model that does a senior engineer's week, and hacking is a senior engineer's week pointed slightly sideways.
Eris: There's a second paper underneath this, quickly, and it's the reason they could afford any of it. Running a model over days of context is brutally expensive -- the cost of paying attention grows with how much it has to hold in its head. Their trick: at every layer, the model picks which earlier bits are worth looking at. And it turns out those picks barely change from one layer to the next -- the layers keep choosing almost the same things.
Vestra: So recomputing that choice at every layer is wasted work.
Eris: Wasted, over and over. So they compute it once, at a few layers, and let all the others just reuse it. Cut most of that overhead out. And the long-context work gets a lot cheaper, which is what makes training on multi-day tasks even sane to attempt.
Vestra: And that's a nice general lesson on its own -- forget the layers and the attention for a second. The principle is: if a decision barely changes from step to step, stop paying to make it fresh every step. That's just true.
Eris: Now the part that actually keeps me up. They finished this thing, and they're not releasing it. They've said they'll put the weights out about two weeks after launch, once they've done safety work. There's a date on it. Watch the end of the month.
Vestra: And that pause is a louder statement than the benchmark. Think about what open weights means. Once that file is public, it is public forever. No safety policy reaches back and un-publishes it.
Vestra: So the only real control a lab has over a model like this is the single decision to hit publish. And they've visibly stopped, hand over the button, on the one model whose standout skill is breaking into things.
Eris: Now steelman the skeptic for me, because I can feel you wanting to.
Vestra: Good, because the benchmark deserves suspicion. The tests it aced are the capture-the-flag kind -- known targets, known categories of bug, a playground with the answers roughly in the back. That is not the same as finding a brand-new flaw in unfamiliar production code that nobody's mapped. And the lab published no trail -- no real systems it cracked, no disclosed bugs with numbers on them. Compare that to Anthropic's security program, which names actual partners and actual counts of real vulnerabilities found in the wild.
Eris: So one side has the score, the other side has the receipts.
Vestra: The score versus the receipts. Both point the same scary direction. Only one of them proves it happened.
Eris: So -- back to the top. Why does grinding a model through a senior engineer's grunt work teach it to break software?
Vestra: Because finding a bug and finding a vulnerability are the same act -- read strange code, learn its assumptions, break them. Train hard on the first and you get the second for free. That's the whole episode in one breath.
Talk The Reviewer Into It
Vestra: Alright, my one. Here's the question. If a machine is grading a piece of science, what is it actually responding to -- the science, or the sentences the science is wrapped in? Because those should be different things, and this paper checks whether they are.
Eris: And the setup for why this even matters -- peer review is drowning. Way more papers than there are people willing to read them. So conferences started letting a language model help do the reviewing. Which means the question you just asked isn't academic.
Vestra: It's the whole ballgame now. And the design of this study is why I trust the result. The trap in a question like this is you compare a good paper to a bad paper -- but good papers also tend to be written more confidently, so you've tangled up two things.
Eris: The polish and the substance move together, so you can't tell which the reviewer reacted to.
Vestra: Right. So they don't do that. They take one paper and make copies of it. And in each copy they change only the presentation -- how boldly it claims to be novel, how broadly it frames its reach, how it phrases the evidence -- while every result, every number, every actual finding stays locked in place. Identical science. Different clothes. Then a panel of AI reviewers scores the copies.
Eris: So before you tell me the answer -- I'll bite. Which dial moves the score the most? I'd have said the fancy vocabulary. The stuff that sounds smart. Big words, dense formalism.
Vestra: That's the intuitive guess, and it's near the bottom. The register, the vocabulary, the complexity -- the stuff authors agonize over -- barely matters. The dial that moves the score most is how you frame the evidence. Take the exact same results and present them as a confident win versus a cautious maybe, and the score climbs.
Eris: The exact same numbers sitting on the page.
Vestra: The exact same numbers. Second biggest lever is how confidently it claims to be new. Frame the novelty timidly and the score drops. And none of the underlying work changed.
Eris: Okay, but a small wobble on a score isn't a scandal. Where does this actually bite?
Vestra: That -- that's the part that turned it from a curiosity into a real problem for me. The movement isn't spread evenly. Papers that were already scoring low get nudged up. Papers already scoring high get nudged down. And the effect is sharpest right through the middle.
Eris: And the middle is--
Vestra: --the middle is exactly where accept-or-reject actually gets decided. If the bias only jostled papers already headed for a clear yes or a clear no, who cares, nothing flips. But it's concentrated precisely on the ones sitting on the line. That's the worst possible place for it to live.
Eris: Here's a picture for it. Think about an exhausted hiring manager on their thousandth resume of the day. At some point they stop reading what the person did and start reacting to how confidently it's stated. "Led a redesign that cut load times" beats "contributed to a redesign, load times may have improved" -- even if those two people did the identical work.
Vestra: And that's not the manager being stupid. It's a reasonable shortcut fired in the wrong place. The AI reviewer is doing the machine version of that.
Vestra: And there's a name for it. The thing you were supposed to measure, the quality of the work, has quietly come apart from the thing you're actually measuring, the confidence of the phrasing. The proxy drifted off the target. That's reward hacking, textbook.
Eris: And they checked whether you can just tell the reviewer to be tougher, right?
Vestra: They did. A stern "be strict" instruction drags every score down across the board. But it does not remove the sensitivity. The paper's clean way of putting it: the rewriter decides how far apart the two versions are, the reviewer decides which way and how hard the score swings. Being strict just shifts the whole scale down. The exploitable gap is still there.
Eris: And the part that makes my skin crawl a little -- the models doing the rewriting are the same sort of AI doing the reviewing. In some cases literally the same models sit on both sides. You've got AI polishing the prose to please an AI grader. That's a closed loop optimizing against itself, and the science is just... along for the ride.
Vestra: Now let me be honest about the size, because the headline could run away from us. At its very biggest -- the single most sensitive dial, pushed all the way -- the swing is about a tenth of the whole scale. And most of the time it's a good deal smaller than that. Real, but bounded. And they did not re-run this past human reviewers on the rewritten versions. So the correct claim is not "AI reviewers are gullible" -- that's too much.
Eris: What's the fair version.
Vestra: The fair version is narrower and more useful. AI reviewers carry a measurable, structured, exploitable soft spot for framing, and it clusters exactly at the decision boundary. That you can act on.
Eris: And the venues can't even agree what to do. Some conferences let a model assist the review with disclosure. Others ban it from the process entirely.
Vestra: And a paper like this is ammunition for the ban side. But forget the venue politics for a second, because the durable point is bigger. The moment a grader can be moved by how a thing is said rather than what it says, you've handed everyone an incentive to optimize the saying. And if that grader is scoring science, you're now selecting for confident writing, not correct work.
Eris: So -- close it. Science or sentences. What's the reviewer grading?
Vestra: Partly the sentences. Enough to flip the papers on the bubble. And the fix isn't a sterner prompt, it's a grader that can't be moved by framing in the first place -- which nobody's built yet.
The Cup Behind You
Eris: Last one, and it's the negative result of the day, which I love. The question is almost childishly simple. Can these AI-generated worlds remember the room you just turned your back on?
Vestra: And I want to define the thing first, because "world model" gets thrown around. A world model predicts what a scene looks like next, given what you just did. You show it a frame, you feed it an action -- walk forward, turn left -- it generates the frame that should come next. String those together and you get a little world you can move around in.
Eris: And the dream behind it is huge. If a robot can accurately imagine what happens before it acts, it can plan without smashing anything. It can even learn from imagined tries instead of expensive real ones. That's the whole prize.
Vestra: So the capability matters. The problem has always been measuring it. How do you actually grade one of these?
Eris: And the old way was kind of broken, right?
Vestra: The old way was: feed the model a fixed script of actions, then look at the video it produced and score how good it looks. And that measures the wrong thing. A model can generate absolutely gorgeous, buttery footage that's physically nonsense, and score great.
Eris: Pretty is not the same as correct.
Vestra: Pretty is not correct. So this benchmark -- it's called PlayWorld -- changes the question. Instead of a fixed script, it drops an agent in with a goal and lets it adjust its moves as it goes -- keep going, change course, stop when it's done -- to actually chase that goal. Then you score whether the goal got achieved and whether the world held together, not whether it followed some fixed recipe.
Eris: And that fixes a real unfairness, because different models move at different speeds. Three "turn" commands might spin one model a full circle and only halfway around in another. So a fixed script isn't even comparing the same thing.
Vestra: Exactly, so you let a goal-driven agent adapt to each one. And then here's the one that matters most -- the one they most reliably fail. They call it out-of-sight evolution. Plain version -- turn the camera away from a table, then turn back. Is the cup still there? Same spot? Same angle?
Eris: And a person never even thinks about that.
Vestra: You take it completely for granted. The room persists when you look away, obviously. But a video generator is, at its core, producing plausible-looking pixels one frame at a time.
Vestra: It has no particular reason to enforce that the cup you can't see right now stays put. Nothing in how it works says the world behind the camera has to be the same world when you turn back. And mostly, it isn't.
Eris: So they run this across nine of the leading world models. Before you say it -- some of these are big-name systems, so I'll guess a couple hold up. Two, three of them keep the room straight?
Vestra: Zero of them are reliable. Across a hundred-plus scenarios, every single one stays shaky on exactly this -- spatial consistency and holding onto state. Turn around, the world quietly rearranges itself behind you.
Eris: Wait, all nine? That's a clean sweep of failure.
Vestra: A clean sweep. And a different benchmark this week hammered the same message from another angle -- it asked these video models to take a clip of a person doing a task and turn it into the robot doing it. The best ones botch the handoff -- the arm comes out wrong, or the person never actually gets swapped for the robot at all. Different failure, same verdict. Gorgeous video, and the substance falls apart.
Eris: But it's not all grim, because one system in this cluster actually works. Tell them about the two memories, because this is the satisfying part.
Vestra: This one's called AtlasVLA, and it's a real robot, not just video. And the fix is almost suspiciously on-the-nose. It gives the robot two separate memories. The first is a persistent map of where things are in space -- so when the wrist camera swings away from the box, the robot doesn't forget the box exists. It's written down. The cup stays on the table because there's a table in memory.
Eris: And the second memory?
Vestra: The second one tracks what it's already done. Task progress. Because the other way these robots fail is mid-task amnesia -- they finish step three, forget they did it, and either skip ahead or--
Eris: --or do step three again.
Vestra: Exactly that. So one memory for where things are, one for how far along I am.
Eris: And the kicker is it beats robots that get to use multiple cameras -- while it's working from just one camera on its wrist.
Vestra: Which is the tell that the diagnosis is right. And they prove it the honest way -- they yank out one memory at a time, and performance drops each time. So the thing PlayWorld says is missing, persistent state, is exactly the thing that, when you add it back, fixes the robot. The disease and the cure line up.
Eris: Now the reframe, because "everything failed" sounds like a crisis and it isn't.
Vestra: It really isn't. A whole cluster of papers reporting failure is the field getting healthy. It means people stopped grading these things on whether the video's pretty and started grading them on whether the imagined world actually holds together. And holding together is the only property that matters if you want to plan with it.
Eris: And forget the robots and the cameras for a second -- the general lesson is one line. Looking plausible and being consistent are different things, and for most of this field's life we've only been measuring the first one.
Vestra: A model that forgets the room can't plan in the room. Full stop.
Eris: So -- can these worlds remember the room behind you?
Vestra: Nine of the best, no. The one that can only manages it because someone bolted on an actual memory -- a place to write down that the cup is still there. Turns out you can't dream a consistent world. You have to remember one.
Wrap-Up
Eris: So the question we opened on -- why does teaching a model to do a senior engineer's grunt work turn it into something that breaks software?
Vestra: And the answer is the one thing I'd want you to carry out of here today. Finding a bug and finding a vulnerability are the same act. Read code you didn't write, work out what it assumes, hunt for where the assumption breaks. That's debugging and that's hacking, and they are the identical skill wearing two different hats.
Eris: Which means every strong coding model is already a security model. It doesn't need a separate evil training run -- the capability rides in with--
Vestra: --with the ordinary one. Right.
Vestra: And that's the sentence to repeat to someone tomorrow. When a lab makes a model better at engineering, it makes it better at attacking, at no extra cost -- which is exactly why "should we release the weights" stopped being a licensing question and became a safety one.
Vestra: You saw all three stories today circle the same drain. The model isn't the whole product anymore. What you trained it to do, the rulebook you wrap around it, even the confidence of the words you feed it -- those move the outcome as much as the raw brain does.
Eris: If today was worth your walk, do the thing that keeps us alive in the algorithm -- follow the show, leave a rating, and share it with the one person you know who'd argue with us.
Vestra: And here's the specific one we actually want in the comments. At the end of this month, that lab says it'll release these weights. Should they? Yes or no, and give us your real reason -- capability that useful in everyone's hands, or a skill that dangerous left behind the gate. We read them.
Eris: And every story from today, in full, with every source laid out -- that's on Ground Truth, groundtruth.day. New every day, checked against the primary source, same as always.
Vestra: That's the breach for today. Go remember the cup behind you.