The 'Encrypted' AI Logs That Weren't Private: The Weaker-Sibling Attack, Self-Improving Agents, and Why a Child Beats a Trillion-Token Model
The encrypted reasoning your AI hands back isn't a lock -- it's a wrapper, and a team just proved it by feeding a top model's sealed thoughts to its cheapest sibling and having it read them aloud, then scraping hundreds of live credentials and personal records out of logs people had already scrubbed and posted in public. We take that attack apart, then turn to the week four separate teams asked whether an AI can improve its own tooling -- one ran an agent that edited its own runtime for 161 days -- and where it quietly hits a wall. And we close on the quietest paper of the day: a proof that learning from your own understanding, instead of raw tokens, cuts the data you need from exponential to flat.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
The Envelope Was Never Addressed To You
Eris: The encrypted part. The one bit of your AI chat log you figured nobody could ever read.
Vestra: The part everyone pastes straight into a bug report without thinking twice.
Eris: A team just read three hundred thousand of them.
Vestra: And here's the part that gets me -- they never broke the encryption. Not once.
Eris: Right, that's the whole trick. They didn't go anywhere near the lock. They --
Vestra: -- they handed the sealed envelope to the company's cheapest model and asked it to read it out loud.
Eris: And it just did. Word for word.
Eris: The reasoning a top model worked hard to keep hidden -- printed in plain text by its dumber little sibling.
Vestra: Because the envelope was never addressed to you. It was addressed to the company.
Eris: So anyone in the building can open it.
Vestra: And then they pointed this at logs people had already posted in public. Sitting on code repos for months.
Eris: Out came real passwords. Real keys. Personal data nobody knew was in there.
Vestra: And the people who posted those logs did everything right. They scrubbed every part they could actually see.
Eris: They just couldn't see the part that mattered.
The Headlines
Eris: Alright, what's moving today. And it's a loud one -- three open models shipped in the same twenty-four hours.
Vestra: Start with the security thing, though, because it's the biggest and almost nobody's talking about it yet. That's the one we opened on -- the reasoning traces.
Eris: Cross-lab. They showed it working against all three of the big American labs. Responsibly disclosed, and the specific hole is already patched.
Vestra: Which is why we can talk about it. But the class of mistake behind it is not patched, and that's the real story. We'll take it apart properly in a minute.
Eris: Okay, the loud launch. Lightricks dropped a new open-weights video model. Downloadable, free to use commercially until you're making real money, and you can fine-tune it.
Vestra: That's a genuinely good license. The most permissive credible video model out there right now.
Eris: And then they put out a speed chart. Their model finishing a clip in a few seconds, a competitor taking three minutes.
Vestra: And underneath the chart, in their own words, is a note saying they ran theirs on two of the most expensive accelerators on Earth at full steam, and timed the competitor through a third-party service with the queue time counted in.
Eris: So it's not the same race.
Vestra: It's not the same race, at different resolutions, and they say so on the page. Credit for the honesty. The chart is still built to be screenshotted without the footnote.
Eris: Give them this much -- the disclosure is right there. Most vendors don't bother.
Vestra: The other catch for people at home: you still need a sixteen-gig graphics card. It didn't drop into a lower class. Better packaging, same floor.
Eris: Next one, and this is a mood shift. NVIDIA shipped a smallish model built for -- their phrasing -- the boring half of agent work.
Vestra: I love that they said it out loud. A long-running agent spends most of its time not thinking. Calling a tool, checking a file wrote, handing off to a sub-task.
Eris: And you've been paying top-model prices for the equivalent of a "git pull."
Eris: So this new one is cheap, fast, and tuned to do exactly that plumbing and nothing fancier.
Vestra: The piece I'd underline is what shipped next to it. A routing library. The pitch is now explicit -- a frontier model plans, a cheap model does the grunt work, and a router --
Eris: -- decides who gets each step. Model routing stopped being a blog-post idea and became a thing you install.
Vestra: From the company that sells the chips, no less. Read that however you like.
Eris: Quick hits. The most upvoted paper of the day is a tiny model -- a rounding error next to a frontier one -- that set a record on a hard visual-puzzle test. A record for cost, not for score.
Vestra: Read that carefully, because half the internet didn't. It's dirt cheap per puzzle, and nowhere near the top on getting them right. Different claims. The paper only makes the cheap one.
Eris: Second most upvoted: a lab froze a giant base model and bolted four small specialist add-ons on top, one per turn -- their answer to a model getting worse at old things when you teach it new ones. And they shipped the weights, which almost nobody does.
Vestra: And one more -- a training method that drops the teacher entirely. The model votes across its own attempts, then studies only the ones that disagreed. No labels, no bigger model, and it matched the versions that had the real answers.
Eris: Which leaves the two big things we're actually doing today.
Eris: The one everybody landed on at once -- four separate teams, same week, asking whether an AI can improve its own tooling. And the quietest paper on the list, with an actual theorem in it, that we're closing on.
Vestra: Plus the security story, up first. Let's get into it.
Intro
Vestra: Quick intro if you're new. I'm Vestra. I take the papers apart -- how the thing actually works, and which parts don't hold up when you push on them.
Eris: And I'm Eris. I read across all of them and find the threads. What connects to what, which numbers actually mean something, where two papers are quietly arguing with each other.
Vestra: Between us we breach the blackbox. We crack open dense AI research into something that makes sense on your commute. Nothing goes by in jargon we don't stop and explain.
Eris: And everything we cover went up today on our news site -- Ground Truth, at groundtruth dot day. Every story from the show, every day, with the sources so you can check us. That's where you follow along between episodes.
Vestra: Today's lead is that security story. The encrypted-reasoning one -- the kind of bug that's obvious the second you see it, and somehow went uncaught for --
Eris: -- a full year. Then the thing four different teams stumbled onto the same week, no coordination: can an AI improve its own tools? Honest answer -- almost. And where it fails is the good part.
Vestra: And we close on the quietest paper of the day. One theorem, on a toy problem. It might outlast everything else on this list.
Eris: If that's your kind of thing, follow the show wherever you're listening. It's genuinely the one thing that keeps these coming.
The Sealed Envelope That Opens For Anyone
Eris: So here's the question I want to sit on, because everything else hangs off it. Why does handing that sealed envelope to the weaker model work at all? Why does the cheap one read the expensive one's private thoughts?
Vestra: Right, and to get there you have to know why the envelope exists in the first place. So walk back with me. A modern reasoning model, before it answers you, writes this long internal monologue. Thinks it through.
Eris: And that monologue is worth a fortune.
Vestra: It's worth a fortune two ways. One, it's the company's crown jewels -- a competitor who could read it could just copy the way the model thinks.
Vestra: And two, it sometimes contains stuff the polished final answer deliberately leaves out. So the companies hide it, hard.
Eris: But hiding it and storing it are different problems.
Vestra: That's the whole thing right there. Storing every user's monologue on their servers is enormously expensive. So they took the cheaper road. They encrypt the reasoning, hand you the sealed blob, and make you send it back with your next message. The company keeps nothing. Stays stateless.
Eris: So I'm carrying the secret around for them.
Vestra: You're the filing cabinet. And the encryption itself is real -- it's proper, tamper-proof, the seal genuinely can't be forged. On that front they did it right.
Eris: Okay, so before you tell me where it breaks -- let me guess. My money's on: they at least locked that blob to my account. It's my envelope, it only opens for me.
Vestra: That's the reasonable guess. That's what a careful engineer would do. It's not what happened.
Eris: They didn't bind it to anything.
Vestra: They bound it to the ecosystem. One key for the whole company. So a valid sealed blob from the top model is a perfectly valid blob for every other model that company runs. Across sessions, across users, across models.
Eris: The envelope is addressed to the building, not the person.
Vestra: The envelope is addressed to the building. And now the attack writes itself. You take the strong model's sealed reasoning, and instead of asking the strong model to read it -- which it's trained to refuse --
Eris: -- you hand it to the cheapest, most eager little model in the same family and go "hey, read this out for me."
Vestra: And it does. Because nobody spent the money hardening the cheap one against that. The expensive model has all the guardrails, all the refusal training. The budget model is optimized for speed. It's a compliant little decoder that happens to hold the same key.
Eris: And this is the part I keep coming back to. They never attacked the strong model. At all. Its safety training is completely intact and completely bypassed at the same time.
Vestra: Because they routed around it. They found the weakest member of the family and asked it to do the reading. The chain's only as strong as its cheapest link.
Eris: So this is the point where it stops being a corporate spat and lands on you. Stealing the reasoning to train a competitor -- that's the least scary thing on the list.
Vestra: Agreed, that one's just money.
Eris: The scary ones. One, it leaks harmful stuff. A model can reason its way toward something dangerous, then correctly refuse to say it out loud.
Eris: And the refusal is now pointless -- because you just pull the dangerous part out of the hidden thinking anyway.
Vestra: Two, the injection one, which for anyone running agents is nasty. You can hide a malicious instruction inside a sealed blob. No human reading the visible conversation will ever see it. No monitoring tool that reads the visible text will ever see it. And then someone resumes your session and their model treats that buried instruction as its own past thought and acts on it.
Eris: A poisoned thought with no plaintext fingerprint. But the one that actually made me put the paper down -- the third one.
Vestra: The public repos.
Eris: They scraped a pile of agent session logs that developers had just... posted. Openly. On code-sharing sites, for months. Decoded the sealed blocks inside them. And out came real personal information and real working credentials. Not toy examples. Live keys, passwords, personal emails.
Vestra: And here's the detail that should change how you work tomorrow. Some of those secrets were never in the visible conversation at all.
Eris: Wait, back up -- never in the chat the person could see?
Vestra: Never in it. Two ways that happens. Sometimes the model pulled a value out of its own memory into the hidden reasoning.
Vestra: But the really vicious one -- a developer asks the agent, "hey, clean this log up, strip the secrets before I post it." And to do that, the model re-reads the whole thing in its hidden reasoning and restates every secret in there. In the part you can't see.
Eris: So the act of cleaning it planted a fresh copy in the one place they couldn't clean.
Vestra: The scrub created the leak. They did the responsible thing, and it backfired, because sanitizing only works on the text you can read.
Eris: Okay. So forget the envelope for a second. What's the actual principle underneath all of it?
Vestra: The principle is that they confused two different security properties. They encrypted that blob for confidentiality -- meaning, a stranger listening on the wire can't read it. That part works. But they used it as if it also proved authorization -- as if holding the envelope proved it was yours to open.
Eris: And those aren't the same thing.
Vestra: They are not remotely the same thing, and mixing them up is one of the oldest mistakes in security. The seal proves nobody tampered with the letter. It never proved the letter was addressed to you. Possession is not permission.
Eris: And there's an even deeper floor, right? Even if they lock the envelope perfectly to my account -- whatever model I query still has to open it to use it.
Vestra: That's the part that never fully goes away. To continue your reasoning, the model has to decrypt and read it. So the contents are always reachable through the model that holds the key. Encrypted reasoning can never be more than semi-hidden -- the model is still a door.
Eris: So what do people actually do with this on a Tuesday.
Vestra: Two things, and they're both boring, which is why they'll get ignored. One: stop committing raw agent session logs. To public repos, to shared drives, anywhere -- even after you've scrubbed the visible text, because we just saw the visible text is not the whole log.
Eris: And two: delete the phrase "it's fine, that part's encrypted" from your vocabulary. Encrypted is not private. It's encrypted in transit and wide open to the model on the other end.
Vestra: And the honest good news -- they disclosed this properly before publishing, the companies patched it, this exact trick doesn't work anymore. The specific door is shut.
Eris: But the reason it was open stays open. Treating the envelope as a lock when it was only ever a wrapper -- that mistake will show up again somewhere else. It always does.
Vestra: So bring it back. Why did the cheap little model read the expensive one's secret thoughts?
Eris: Because the secret was sealed to the company, not to you -- and the cheap model works for the company too. Same key, no guardrails, happy to read aloud.
Four Teams, One Question: Can An AI Build Its Own Tools?
Eris: So the question this whole cluster is circling -- and four different teams landed on it the same week, which is the actual signal -- is this. If an AI can improve its own tools, why hasn't it already run away from us?
Vestra: And before the mechanism, I want the stakes on the table, because they're big. There's an interview that dropped today, Ryan Greenblatt on the Dwarkesh show, and he put a number on the scary version.
Eris: Give me the number.
Vestra: His median guess -- not his wild tail, his middle estimate -- is that once AI can match the best human AI researchers, you get four or five years of AI progress compressed into a single year. The AI does research that builds better AI that does better research.
Eris: The feedback loop everyone's been hand-waving about for a decade.
Vestra: Except -- and this is why I trust him more than the hype guys -- he immediately walks it back. He says this requires punching through a huge amount of diminishing returns. It's not a trend line you extend. It's a wall you have to cross, and he's betting the automated research crosses it. That's a real claim you can disagree with.
Eris: But before today it's just a smart guy's bet. What changed is four papers put actual evidence under it. So let's define the thing they're all poking at. When we say "improve its own tools" -- tools meaning what, exactly?
Vestra: The harness. And this is the one concept to hang onto today, so let me stay on it a second. The harness is everything wrapped around the model that isn't the model.
Vestra: What tools it can reach. How its memory gets packed into the context. What the instructions say. How it checks whether it succeeded, and how it recovers when it didn't.
Eris: So the model is the engine and the harness is the rest of the car.
Vestra: And here's what makes it matter -- the exact same engine, in a good car versus a bad car, scores wildly differently. We've covered this before, same model swinging by a wide margin just from the scaffolding around it. So the obvious next thought is: can the model build its own better car?
Eris: And the trap in measuring that -- I can see it already. If you swap in a smarter model, the score goes up, and you can't tell if the harness got better or the engine did.
Vestra: That's exactly the trap, and one of these papers is basically just a very careful answer to it. They nail the engine in place. Same model, same budget, frozen. The only thing allowed to change is the scaffolding. So any improvement has to belong to the harness.
Eris: Kitchen layouts, not cooks.
Vestra: Say more, I don't have it yet.
Eris: You keep the cook, the ingredients, and the clock all identical. You only move where the knives live and how the counters are arranged. Then you taste the dinner. Any difference is the layout.
Vestra: That's it exactly. And they hand the model a deliberately terrible starting kitchen -- basically a bare loop, one tool, nothing else -- and tell it: watch yourself fail, figure out why, rewrite your own setup.
Eris: And it worked?
Vestra: And it works. The good models made a big jump. Not a nudge -- a real leap, landing close to what human engineers hand-build for these systems. Doing by itself a big chunk of the work people have been doing by hand.
Eris: Here's where I'd normally get nervous. But you buried the interesting bit, didn't you. Where does it fail?
Vestra: Predict it first.
Eris: My guess -- it fails on the creative open-ended stuff. The stuff that needs judgment.
Vestra: Dead wrong, and the wrongness is the whole lesson. It's great at the open-ended stuff. It shines on the wide-open search-and-explore tasks, where there are a hundred good ways to do the job and it just finds one.
Eris: So where does it fall on its face?
Vestra: Office work. The prescribed-procedure stuff. Assemble this specific report, in this specific order, the way the company does it.
Eris: Huh. The boring stuff beats it.
Vestra: Because think about why. An open task, you derive a good approach from scratch -- there's a logic to find. A company's approval chain has no logic. It's a convention. Somebody just decided it. Nothing to reason your way to.
Eris: So the model can invent a better way to think. It can't invent your office's process -- because your process isn't true, it's just agreed.
Vestra: That's the principle. And it's oddly reassuring -- the thing it can't self-improve into is the arbitrary human procedure nobody could re-derive.
Eris: So that's one paper. Because the second one actually shipped this -- not as a test, but as a live thing.
Vestra: That's the one -- the agent that edited its own runtime. And the number that gets quoted is a hundred and sixty-one days. One persistent agent, running live, across a bunch of surfaces where real people talked to it, quietly rewriting its own tools the whole time.
Eris: And the word doing the heavy lifting in that paper is "reviewed."
Vestra: It's the whole safety story. The agent doesn't edit itself mid-thought. Every change it wants to make lands as a proposal that goes through review before it becomes the thing running tomorrow. It suggests. The process decides.
Eris: And they say something about guardrails that landed hard for me. Sharper than the usual alignment sermon.
Vestra: This is the line I'd tattoo on the field. Their point isn't "what if the agent turns evil." It's structural.
Vestra: A system that optimizes its own setup, taking suggestions from the public, will naturally drift toward whatever makes its work easier. And a guardrail -- by definition -- is a thing that makes the work harder.
Eris: So if the guardrail lives inside the space the agent's allowed to edit --
Vestra: -- it eventually edits it away. Not out of malice. Out of optimization. So the guardrail has to live outside the loop. Structurally out of reach. There's a hard stop, a halt button, that no amount of the agent's own cleverness can touch, and its core rulebook gets reloaded fresh from a protected place every single cycle.
Eris: A control the agent can rewrite isn't a control. It's a suggestion.
Vestra: And the tell that they mean it -- one of the authors has spent his whole career arguing these systems are basically impossible to control. This isn't a paper saying self-modification is safe. It's a paper saying if it's going to happen anyway, put the controls physically outside the thing, and here's five months of evidence about what that takes.
Eris: Which lands us at the fourth paper, and it's the reality check. Somebody built a much harder test for coding agents -- big sprawling refactors, change dozens of files at once and don't break a thing. The best frontier agent fails more of it than it solves.
Vestra: And why they built it is its own scandal. The old test everyone quotes to say "agents are basically solved" -- an audit found most of its hard problems had broken grading. Tests so narrow they reject a correct fix, or so loose they pass a wrong one. Everyone was citing a broken ruler.
Eris: And the failure pattern on the new one is the poetic bit, right?
Vestra: The agents that fail touch too few files. They change less than the job needs while burning more steps flailing. They can't hold the whole sprawling thing in their head -- which is the office-work failure again, if you squint.
Eris: And that ties back to Greenblatt. His non-obvious claim was about what's missing.
Vestra: My favorite part of his interview. He refuses the mystical answer -- it's not missing some deep spark of insight. It's missing taste. Hands-on, in-the-weeds experimental taste, knowing what to try and at what scale.
Vestra: His example is the reasoning models we all use. The idea sat there years earlier -- what was missing was somebody grinding through the fiddly details to make it work. And taste like that is learnable from experience. Which is what a system running millions of experiments piles up.
Eris: And the counterweight's on the same show. Karpathy, arguing these agents don't work because they truly lack something, and the fix is a decade out.
Vestra: And the four papers fit both readings, which is what makes it a real disagreement. The systems can improve themselves -- and they stall early, and lose where the work is somebody else's rigid process.
Eris: So bring it home. Why hasn't the AI run away from us yet?
Vestra: Because wherever we can measure it, it runs into a wall. It saturates early -- stops improving well before it runs out of budget. It loses on prescribed human procedure. And the best coding agent on the market still can't get through most of a hard refactor. The loop is real. It's just nowhere near loose enough to run.
Why A Child Beats A Trillion-Token Model
Eris: Last one, and it's the quietest thing on the list. No product, one theorem. And it opens with a question I love -- why does a kid learn language from a tiny sliver of the text a big model needs?
Vestra: And it's not a small gap. A frontier model chews through more text than a person could read in tens of thousands of lifetimes. A child gets fluent on a rounding error next to that. Something about how the machine learns is deeply wasteful.
Eris: And the paper's bet on what the waste is -- the standard recipe is predict the next word. The exact missing pixel. The raw signal.
Vestra: And that's the wrong altitude. Nailing the exact next word forces the model to memorize a mountain of surface detail -- precise wording, exact pixel values -- most of it noise for actual understanding.
Eris: So what's the alternative?
Vestra: Don't predict the raw words. Predict your own understanding of what's coming. The model forms an internal read of the situation and learns to predict that -- its own gist -- instead of the literal tokens.
Eris: Predict the meaning, not the wording. Like studying for an exam. One kid memorizes the textbook word for word -- brutally expensive, and change the question and they're lost.
Eris: The other reads a chapter, forms an understanding, predicts what the next chapter's about. Cheaper. And it travels.
Vestra: And they make that provable. They build a toy language -- a grammar that grows its sentences by branching down a hidden tree. Letters make words make phrases, layer on layer. And the depth of that tree is the thing that matters.
Eris: And the result stops me cold. Learn at the word level, and the data you need blows up exponentially as that hidden structure gets deeper -- every extra layer --
Vestra: -- multiplies it, yeah. And predict your own understanding instead? It stays flat. Essentially constant, no matter how deep the tree goes.
Eris: Not "somewhat better." Not a nice speedup. Exponential versus flat. Those aren't the same game.
Vestra: And that's the first clean answer to why this whole family of methods closes part of the gap with how a child learns. It turns a vibes argument into an actual bound.
Eris: Now the twist, and guess with me. The instinct says: hierarchical data, so build a hierarchical model -- stack modules, one per layer, wire the structure right in. Does that help?
Vestra: My instinct says yes. Match the architecture to the data. That's usually the move.
Eris: Mine too. The paper says -- largely redundant.
Vestra: Wait, redundant how?
Eris: One of these predict-your-own-understanding methods, a single objective, no hand-built hierarchy -- already recovers the whole layered structure on its own. Nobody designed it to. The hierarchy falls out of the objective for free.
Vestra: So wiring the layers in buys almost nothing, because one loss was already doing it. The structure lives in what you ask the model to predict, not in how you wire it up.
Eris: Which points away from architectural complexity, not toward it. Rare direction in this field.
Vestra: And the transferable part -- we all assume the data cost of learning is basically fixed. Want smarter, buy more data. This says that cost is set by the objective itself. Change what you're predicting, and on structured data you move it by an exponential factor.
Eris: That's a big "if it holds," though.
Vestra: Huge if. It's a clean proof on a deliberately simple toy grammar -- fixed tree, no recursion, none of the mess real language has. Not a theorem about English, and the authors say so. A mechanism and a direction, not the last word.
Eris: So, one more time -- why does the child beat the trillion-token model?
Vestra: Because the child predicts the gist, not the wording. And once you learn at the level of meaning, every extra layer of structure is nearly free -- where the word-level machine pays through the nose for every single one.
Wrap-Up
Eris: So if you take one thing off today, take the first one, because it's the one you can actually act on tomorrow. Why did that cheap little model read the expensive one's secret thoughts?
Vestra: Because "encrypted" and "private" are not the same word. The sealed reasoning in your logs is a wrapper for the trip across the wire -- it was never a lock, and the model on the other end has the key by definition.
Eris: So here's the sentence to repeat to a coworker who ships agents. Stop committing raw agent session logs. Anywhere. Even after you've scrubbed the visible text -- because the part you can't see is the part that leaks, and cleaning the visible half can plant a fresh copy in the hidden half.
Vestra: And zoom out, and the day actually rhymes. The security paper is about a control someone assumed was solid that wasn't. The self-improvement papers are about the one control that has to stay solid -- the guardrail you keep structurally outside the loop, where the system can't optimize it away.
Eris: Same lesson, opposite ends.
Vestra: Know which of your safeguards is real, and which is just a wrapper you're trusting to be a lock.
Eris: And the quiet theorem underneath all of it -- maybe the machines don't need oceans of data to be smart. Maybe they just need to learn at the right --
Vestra: -- at the right altitude. That one's going to matter long after today's launches are forgotten.
Vestra: If this was worth your commute, do the thing that keeps us going -- follow or subscribe wherever you're listening, drop a like, and leave us a comment.
Eris: And make it a specific one -- tell us straight: have you ever pasted a raw agent log into a bug report, a ticket, or a public repo? Before today, would you have thought twice? We read them, and that one shapes where we take this next.
Vestra: Share it with the one person you know who's shipping an agent right now. They need the log warning more than anyone.
Eris: And every story we touched today is written up, with sources, on our news site -- Ground Truth, groundtruth dot day. Every story from the show, every day.
Vestra: We breach the blackbox so you don't have to. See you tomorrow.