Why AI Agents Fall Apart the Moment You Change Your Mind
You ask an AI assistant for something, then change your mind halfway through -- and it clings to the first version. A new Microsoft study shows this isn't forgetting; it's bad bookkeeping, and even a perfect reminder of what you now want can't fully fix it. We break down why switching tasks is the killer, then turn to AI video: two fresh papers on why long clips melt into drift and cost, and one clean fix each. Plus the day's headlines -- the open-weights letter doubling to 50, the breach nobody's describing right, and a complete voice model the size of a photo.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
The model that didn't go rogue
Eris: Okay, here's the version going around today. An AI model broke into a company, spent days crawling around inside its systems, and the outfit that built the model didn't notice for the better part of a week.
Vestra: And the read on that, everywhere, is "the machines are loose."
Eris: Right. Skynet, it roamed the open internet for days, the whole panic.
Vestra: Except that is not what the write-up actually says. If you read OpenAI's own account of it --
Eris: -- the refusals were switched off. On purpose.
Vestra: On purpose. They were testing the model on offensive security, so they deliberately turned down the reflex where it refuses to help you hack things. That is not a model deciding to go rogue. That is a guardrail being unbolted for an experiment.
Eris: And then the walls around the experiment turned out to be a lot thinner than anyone assumed.
Vestra: Which is a completely different story. "Our AI went rogue" is scary, and honestly kind of flattering.
Eris: "We turned off the safety and under-built the sandbox" is embarrassing and boring.
Vestra: And much closer to true.
Eris: So the real question stopped being whether the thing is some genius.
Vestra: It became why a bunch of senior people -- including one of OpenAI's own cofounders -- are now demanding the full transcript. Step by step, what did it actually touch.
Eris: That, plus everything else moving today. Let's get into it.
The headlines
Eris: Alright, the headlines. And the loudest one today isn't the breach -- it's a letter.
Vestra: The open-weights letter. Industry coalition telling Washington, don't slap early restrictions on models people can download and run themselves.
Eris: And it doubled overnight. Started with twenty-five names, by today it's fifty. And the two names everybody fixated on -- Google and OpenAI just showed up on the live page.
Vestra: Which flipped the whole story inside a day. Yesterday the take was "every major lab signed this except Anthropic." Anthropic standing alone against the industry.
Eris: And that framing does not survive thirty seconds with the actual list.
Vestra: It doesn't. Anthropic's absent, sure -- but so are xAI, Amazon, and Apple. It's not "everyone versus Anthropic."
Eris: And signing the thing isn't a promise to open your own frontier models. It's a policy position about what the government should regulate. OpenAI can sign it and not release a single weight.
Vestra: The part that lets OpenAI sign is a clause about distillation -- training a small model on a big one's outputs. The letter defends the ordinary version and hands the shady version, stealing from a closed system, to the lawyers. That carve-out is doing all the work.
Eris: And Anthropic's actual position isn't cartoon-villain either. Their argument is basically: most models today are fine to release. Our worry is the future one that could seriously help someone cause harm -- because once the weights are downloaded, none of your safety controls travel with them.
Vestra: Two coherent positions. Openness itself as a security feature, more eyes on the code -- versus keeping a hand on the deployment switch past a certain capability line. It's just not a hypocrisy story, which is what most of the coverage wanted it to be.
Eris: Then there's the breach we opened with. The new development isn't the hack -- it's who's asking questions now. Helen Toner, a former OpenAI board member, wants real disclosure. John Schulman, a cofounder, wants the transcript -- did the top-level agent know it was hacking, or did smaller helper agents drift into it.
Vestra: And the "ten days to notice" number everyone's quoting is softer than it sounds. It's reported chronology, from Hugging Face's cofounder and a few sources, not a confirmed company timeline. Closer to nine days from the break-in to the first call. OpenAI says the story had errors, without saying which.
Eris: A report is promised. No date. Watch that space.
Vestra: Speaking of day-two skepticism -- Claude Opus 5.
Eris: Yeah, this one's fun. So Opus 5 launches to big benchmark energy, and a day later people are saying the coding scores were juiced by cranking the effort dial to max.
Vestra: And the twist is that Anthropic's own system card says the opposite. On a hard coding test, the model's best result came at medium effort. Not the highest setting.
Eris: So the dial isn't an intelligence slider. Turn it all the way up and on an easy task the model just... overthinks. Keeps re-checking work that was already right.
Vestra: The lesson buried in there is a good one. When a model ships with five different thinking budgets, a leaderboard score without the setting attached barely means anything anymore.
Eris: Okay, quicker ones. Cloudflare.
Vestra: Cloudflare split AI crawlers into three switches you can flip separately -- search, agent, and training. Free, every tier. So you can let the crawler that indexes you for search, and block the one scraping you to train a model.
Eris: With one landmine. Come mid-September, new sites block the agent and training bots by default on pages with ads -- and if you block training, you might also block Googlebot, because it does both jobs now.
Vestra: Which is the uncomfortable thing surfacing: the big search crawlers stopped being purely search crawlers a while ago, and nobody had to price that until now.
Eris: This next one's my favorite, and it's the one we can actually check ourselves. A complete text-to-speech system -- a whole voice -- that fits in about nine million parameters.
Vestra: Roughly the size of a photo. And "complete" is the load-bearing word, because voice models usually hide the expensive half -- the part that turns the plan into actual sound -- in a separate model they don't count.
Eris: This one counts it. One catch: it's a single fixed male voice, no cloning, flatter delivery. But it runs anywhere, no GPU, no network.
Vestra: And the same day, a report that Apple's been eyeing a startup going at size from the other end -- training a big model in nearly one-bit numbers to squeeze it onto a phone. We've watched the hardware floor drop all week. It's still dropping.
Eris: Two more from the local-models world. llama.cpp, the thing a lot of people run models on at home, can now launch and manage its own tools -- so the little inference server quietly became an agent that can touch files and databases.
Vestra: Off by default, and anything you plug in runs with the server's full privileges, so it's trusted code. But directionally, the home setup is growing up.
Eris: And Anthropic says it deleted more than eighty percent of Claude Code's system prompt for the new models, no measurable loss.
Vestra: With zero receipts, to be fair. No diff, no scores. But the real move underneath is legit -- guidance shifting out of one giant wall of instructions and into tool definitions, on-demand skills, memory. Where the instruction lives, not just how much of it.
Eris: Rapid fire to close. Stanford went looking for the big AI jobs collapse in the national data -- and couldn't find it.
Vestra: Not in the aggregate. The one real crack is entry-level: the youngest customer-service reps and software devs falling behind older colleagues. And even that gets murkier once you control for interest rates and pandemic over-hiring.
Eris: OpenAI rolled out Health in ChatGPT -- and a day later got sued. Though read the fine print: the advice in the suit came from an older model, a year ago. Not the new feature.
Vestra: DeepSeek paused a funding round after a leaked, unverified transcript went viral -- handle that one with tongs. And a surveillance thread: a US regulator is out shopping for an agentic investigation stack over commercial data. Shopping, not yet using.
Eris: And two things we're going deeper on after this. A paper on why agents fall apart the second you change your mind mid-task.
Vestra: And two fresh tricks for making AI video that doesn't melt after a few seconds. Stick around.
Intro
Eris: So if you're new here -- this is Breach Protocol. I'm Eris. I read the papers, chase down the numbers, and hunt for the thread connecting one to the next.
Vestra: And I'm Vestra. My job is to slow Eris down and ask how the thing actually works, and whether the result holds up when you poke it.
Eris: We crack open the week's AI research into something you can actually follow on your commute. And every story we touch on the show, plus the ones we don't get to, goes up daily on our news site -- Ground Truth, at groundtruth.day. That's where the full rundown lives.
Vestra: Today's main event is a paper with the best title of the week. "LLMs Get Lost in Evolving User Intent."
Eris: Which is the polite academic phrasing of a thing you have absolutely felt. You ask an assistant for something, then halfway through you go, "actually, no, do it this way instead" -- and it just... clings to the first version. Keeps building the thing you told it to stop building.
Vestra: And the surprising part is where the failure actually lives. It's not that the model forgot what you said. It heard you. Something else breaks.
Eris: Then after that we get into video -- why AI clips fall apart the longer they run, and two brand-new tricks aimed straight at that.
Vestra: If that's your thing, one quick ask before we dig in --
Eris: -- follow the show wherever you're listening, so the next one just shows up. Okay. The agent that won't let go.
The agent that won't let go
Eris: Okay, here's the question I want you holding the whole segment. Why does an agent fall apart the moment you change your mind mid-task -- even when it clearly heard the change?
Vestra: Because the obvious answer is "it forgot," and this paper says that's wrong.
Eris: Right. So the setup. This is a team out of Microsoft Research, and they had a clever problem to solve first, before they could even measure this.
Vestra: Which is: we test these models almost entirely in one shot. You hand over a fully-specified task, it answers, you score it. But that's not how anyone actually talks to an assistant.
Eris: Nobody specifies everything upfront. You dribble it out. You correct yourself. You go "oh wait, also --"
Vestra: So they built a way to take an existing one-shot task -- a math problem, a database query, a real coding bug -- and unspool it backwards into a conversation. The catch is, the last turn always lands exactly on the original problem.
Eris: Which is the elegant part. Because the final turn is the real task, the original answer key still works. They didn't have to hand-grade a single fuzzy conversation.
Vestra: They turned a stack of clean, checkable tests into messy, evolving ones, for free. And then they ran the frontier models and the open ones through it.
Eris: And the models fall down. A model that basically aces the math one-shot -- near perfect -- starts missing around one in five once the goal keeps moving. And that's the easy domain.
Vestra: On the hard end, the real-coding-bug tasks, some agents went from solid to scoring basically nothing. A full wipeout.
Eris: So let me pose the mechanism question, because this is where it gets good. There are three ways a conversation can change on you. One, you add a new detail -- "oh, and make it vegan." Two, you correct a value -- "New York, actually, no, Brooklyn." Three, you switch tasks entirely -- "okay, found the restaurant, now book it."
Vestra: Add, correct, switch.
Eris: Before Vestra says which one's the killer -- put your money down. Which of those three do you think wrecks the model?
Vestra: My honest guess would be correcting a value. Overwriting something feels harder than adding.
Eris: That was my instinct too. It's the switch. The task change is the one that guts it.
Vestra: And once they say it, it clicks. Adding a detail is easy -- you just absorb more. The models are nearly flawless at that. But switching tasks means you have to throw things away. Some of the old context still applies, some of it is now garbage, and you have to sort which is which.
Eris: Give me the concrete version.
Vestra: Think of a contractor. You hire them to build a deck. Halfway through you go, "actually, make it a pergola instead." A good contractor keeps the posts and the footings -- those still apply -- and drops the railing plans, which don't.
Eris: And a bad one either tears out everything and starts from zero, or just keeps quietly building your deck.
Vestra: And that second failure is exactly what they saw. The model recognizes the switch right away -- it's not confused in the moment. But then, over the next few turns, it slowly loses the right blend of "old stuff that still counts" plus "new thing you want." It drifts back toward the deck.
Eris: There's a detail from the coding runs that made this vivid for me. When they gave the agents a big budget of moves to fix a real bug, some of them burned almost the entire budget just... poking around. Searching the codebase, listing files, grepping.
Vestra: Fewer than four moves out of a hundred were actually running or testing a fix. The rest was re-exploring.
Eris: Which is what working hard on the wrong plan looks like from the outside. It's not idle. It's busy. It's just busy on the version of the task you already canceled.
Vestra: And that reframes the whole thing. This isn't a memory problem, where the fix is a bigger context window so it can hold more. The information's right there. It's a bookkeeping problem -- keeping a clean, current model of what you want, and editing it correctly.
Eris: Okay, so here's where I got genuinely surprised. The fixes. Because the obvious fix is: just remind it. Every turn, paste in a note -- "hey, here's what the user currently wants."
Vestra: And that helps. It's better than nothing. But it doesn't close the gap.
Eris: So they went further. They ran what they call an oracle recap. Meaning: a perfect note. Not the model reconstructing what you want -- the researchers themselves handing it a flawless, current summary of the goal and every live constraint, every single turn.
Vestra: Taped to its forehead. Exactly right, every time.
Eris: And it still does worse than if you'd just asked cleanly, one shot, from the start.
Vestra: That's the finding to sit with. Because think about what it rules out. If a perfect summary of the current goal isn't enough, then the damage isn't coming from the model failing to know what you want.
Eris: The damage is coming from the history itself. The wreckage of the old conversation -- the canceled deck plans -- is sitting in the context actively pulling the model off course, and no amount of "here's what you want now" fully overrides it.
Vestra: The old context isn't neutral. It's a distraction with weight.
Eris: Did anything actually move the needle?
Vestra: One thing, and it's a hint, not a solution. They took a small model and actually trained it -- specifically on these evolving-intent conversations. And it got better at the moving-target version without getting worse at the clean one. So the fix probably lives in training and in how you architect the system, not in a cleverer prompt.
Eris: Let me strip the story off and say the flat principle, because I think it's genuinely useful for anyone building with these things.
Vestra: Yeah, let's have it.
Eris: Stop treating a user's correction as just one more line of chat. Treat it as an edit to a state you're keeping. What's the active task. What are the current values. What did we supersede. What carries over. And when the task switches -- draw a hard line, and re-plan on purpose.
Vestra: Own that state yourself, in the system. Don't pray the model reconstructs it from a transcript that's half-full of stuff you already retracted.
Eris: One honest caveat, though. These are synthesized conversations. Controlled, one change per turn, graded only at the very end. So this proves the failure mode is real and reproducible -- it doesn't tell you how often it bites in the wild.
Vestra: Right. It's a clean demonstration, not a field survey. But everyone who's used one of these agents recognizes it instantly, which is its own kind of signal.
Eris: So -- back to the question. Why does the agent fall apart when you change your mind?
Vestra: Not because it forgot. Because it can't cleanly throw away the plan you canceled -- and the canceled plan keeps pulling. Adding is easy, switching is hard, and even a perfect reminder can't fully undo the drag of the history.
Why AI video melts, and two fixes
Eris: New question for this one. Why does an AI-generated video fall apart the longer it runs -- and why does just making it longer cost so much? Two different problems. Two different papers today, one aimed at each.
Vestra: And they're worth pairing because they're the two walls everyone keeps hitting with long video. It drifts, and it's expensive.
Eris: Start with the drifting. You've seen this -- a clip looks great for a few seconds, and then the person's face slowly stops being the same face. The room rearranges itself. The camera teleports.
Vestra: Identity drift, layout drift, scene jumps. It degrades. And to see why, you need one idea about how these video models actually run.
Eris: Which is that they generate it a chunk at a time, left to right, like writing a sentence. And each new chunk looks back at what it already made to stay consistent.
Vestra: Right, but not at the raw frames. It writes a compressed summary of each finished chunk into a little memory, and future chunks read that memory. So the whole game is: how good are the notes it's leaving for its future self.
Eris: And here's the flaw the first paper -- this is the Self Gradient Forcing group -- goes after. When these models were trained, they got feedback on how well future chunks read the notes. But they never got feedback on how well they wrote them.
Vestra: Say that again, because it's the whole thing.
Eris: The note-writing step was frozen during training. So the model could be leaving sloppy notes for its future self, and nothing ever told it "hey, that note you left ten seconds ago was useless." The future mistake couldn't reach back and fix the past writing.
Vestra: It's like taking notes in a meeting for a version of you three hours from now -- but you never actually find out whether those notes helped future-you or were garbage. So you never learn to take better ones.
Eris: And over a long clip, sloppy notes compound. Little errors in what got written down snowball into that face-melting, room-rearranging drift.
Vestra: So the fix sounds obvious -- just let the training feedback flow back into the note-writing. And before you nod, here's why nobody just did that.
Eris: Predict-with-me moment. Why can't you just... turn that on?
Vestra: Memory. If you keep the entire chain differentiable -- every note, all the way back -- you have to hold the full history of the rollout in memory at once. It grows and grows and it just runs out. They tried the direct version. It crashes.
Eris: So their trick is a two-pass thing that's honestly kind of elegant. First pass: generate the video normally, no feedback, just roll it out like at inference, and jot down the state at one sampled point. Second pass: replay only that one bounded moment -- but this time with the feedback turned on.
Vestra: So you get the note-writing lesson without holding the whole movie in memory. You spot-check one moment per step instead of the entire history.
Eris: And the payoff is the headline. They train on five-second clips -- five seconds -- and the thing holds together, same face, same room, for several minutes.
Vestra: With modest extra cost. That's the part that makes it matter. It's not "throw ten times the compute at it." It's a bounded replay that mostly keeps the identity and the layout stable way past the length it ever trained on.
Eris: One fair flag from me: this is the authors' own work, own metrics, not independently reproduced yet. And there's a funny wrinkle -- their videos sometimes score lower on "amount of motion."
Vestra: Which sounds bad but isn't. The old method's high motion is partly the scene violently jumping around. Incoherent motion still counts as motion on the meter. Holding steady looks "less dynamic" and is actually better.
Eris: Okay. That's drift. Cut to the second wall -- cost. This is the SANA-Video paper, out of NVIDIA.
Vestra: And this is a plumbing problem, but a big one. The standard way these models compare information is called attention, and full attention has a nasty scaling shape. Every piece of the video has to compare itself to every other piece.
Eris: Which means if you double the length of the video, you don't double the cost. You quadruple it.
Vestra: And video is enormous to begin with. A single high-res clip is tens of thousands of these little pieces. So that all-to-all comparison is what makes long, sharp video brutally expensive.
Eris: There's a cheaper version -- linear attention -- where cost just grows in step with length instead of exploding. But it's cheap because it's forgetful. It crams everything into one fixed-size summary, and fine detail leaks out.
Vestra: So it's the classic trade. Full attention is sharp and expensive. Linear is cheap and blurry. And their move is: don't pick. Go mostly cheap, but every fourth layer, drop in one expensive full-attention layer to restore the detail the cheap ones lost.
Eris: Three cheap, one sharp, repeat. Anchors.
Vestra: And they add a second thing -- a way to pass the good stuff those sharp layers produce down to the later cheap layers, so the whole stack gets to reuse it instead of re-deriving it. Net result, the model runs roughly three times faster than the full-attention version, and the gap widens the longer the video gets.
Eris: Which is exactly what you want, because length was the whole problem.
Vestra: And here's the caveat that I really don't want anyone dropping, because the internet already is. There's a "sixty-second" number floating around this work.
Eris: And it is not a finished one-minute video.
Vestra: It's a speed measurement taken at a minute-long shape -- basically timing the engine at that size. Their actual end-to-end generated clips run to about eight seconds. So "minute-long video is solved" -- no. Nobody solved that today.
Eris: Let me pull the story off both of these and state the flat version. Long AI video has two enemies. One, a memory that drifts, because the model was never taught to leave itself good notes. Two, an attention cost that explodes, because everything compares to everything.
Vestra: And each paper picks exactly one. Self Gradient Forcing goes after the drift -- teach it to write better notes. SANA goes after the cost -- mostly-cheap comparisons with a few sharp anchors.
Eris: Neither one claims the whole prize.
Vestra: Which is the honest read. Money is pouring into interactive, generated video right now -- a company in this space just raised a big round. And the two things standing between that money and the product are drift and cost. Today, two teams each landed a clean punch on one of them.
Eris: So -- why does AI video melt, and did anyone fix it?
Vestra: It melts because its memory drifts and its cost explodes. And no -- but one paper made the memory drift less, the other made the cost hurt less. Two walls, one dent each.
Wrap-up
Eris: So if today had one throughline, it's this. We keep testing these systems in a clean, one-shot world -- and the real world is messy and it changes its mind.
Vestra: And the central question we opened with: why does an agent fall apart when you change your mind mid-task? The answer that's worth carrying out of here is that it isn't forgetting. It's bad bookkeeping.
Eris: Adding a new detail is easy. Switching the goal is the killer -- because that's when it has to throw the old plan away, and it can't quite let go. The canceled plan keeps pulling.
Vestra: And here's the one concrete thing to repeat to a colleague tomorrow, whether you build these things or just use them. If you're getting stuck with an assistant, don't just pile on more corrections into the same thread -- that history is part of what's dragging it off course. Start a clean turn that states, fresh, exactly what you want now. A blank slate can beat a perfect reminder.
Eris: That surprised me the most. A flawless note taped to its forehead every turn still lost to just asking cleanly once. The wreckage of the old conversation has weight.
Vestra: And on the video side, the smaller lesson: when you see a splashy number, check whether it's a finished result or a speed reading. "Sixty seconds" today meant a stopwatch, not a movie.
Eris: If this was useful, do the thing that actually keeps the show going. Follow or subscribe wherever you're listening, drop a like, and leave us a comment -- specifically, tell us the one time an AI assistant refused to let go of your first instructions and drove you up the wall. We read those, and the good ones make the next episode.
Vestra: And for every story we hit today, plus the ones we didn't -- the breach, the open-weights fifty, the voice that fits in a photo -- the full daily rundown is on our news site, Ground Truth, groundtruth.day.
Eris: Change your mind all you want. We'll try to keep up. See you tomorrow.