Brain-Like Modules Inside LLMs, and Why a Small Model Advises as Well as a Flagship
Delete the right neurons in a large language model and its grammar collapses while its physics stays perfect -- delete a different set and the sentences stay beautiful while the answers go wrong. MIT found six models had quietly sorted themselves into brain-like modules, and a second team traced the giant 'anchor' numbers inside every model to a single layer. Then the flip side: a small open model writes agent upgrades as good as a frontier flagship's, and the weak models still can't follow them. What a model does lives less in which model it is than in the structure inside it and the scaffolding around it.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
You Can Give a Model Brain Damage
Eris: You can give a language model brain damage. On purpose. Reach in, delete a specific set of neurons, and watch what breaks.
Vestra: And the part that breaks is the tell. That's the whole reason this landed.
Eris: Right -- so a team at MIT does exactly this. They find the neurons a model uses for language.
Eris: The actual grammar-and-fluency ones. Then they delete them.
Vestra: And you'd expect the whole thing to fall over--
Eris: --It doesn't. It keeps the physics right. Ask it about a stretched rubber band and it still reasons correctly -- it just can't say it. The sentences come out mangled. Misspelled, broken grammar.
Vestra: "The polymer chain network that form its structure." Correct idea, broken mouth.
Eris: Then you do the opposite. Leave the language alone, delete the physics neurons instead.
Vestra: And now it's the mirror image. Beautiful, fluent, confident sentences -- saying something flat wrong.
Eris: Grammar intact, answer inverted. Form and content, coming apart under the knife.
Vestra: Which is a thing we see in people. Certain brain injuries do exactly that -- fluent speech, no meaning. Or meaning intact, no speech.
Eris: And that's the part I want to sit with. Nobody built the model this way. It sorted itself into departments. So why would a machine that pays no price for using more neurons ever bother to specialize?
Vestra: And whether "it looks like a brain" is a real finding or just a flattering picture. Because that distinction is the entire episode.
The Headlines
Vestra: Okay, the headlines. And the modularity paper is the one everyone's going to be chewing on, so we're coming back to it properly later.
Eris: We are. But it's not even the only interpretability result this week saying the same thing. There's a second one -- somebody found the single layer inside these models where those giant weird numbers come from. The ones everybody wrote off as--
Vestra: --noise. The massive activations. Yeah, that's a real pairing. Two teams, one week, both saying: the thing you dismissed is load-bearing. We'll get into that.
Eris: The other big cluster today is agents. Specifically, agents that rewrite their own instructions. And there's a finding I did not see coming -- a small open model, a fraction the size of a frontier flagship, writes upgrades for an agent that are just as good.
Vestra: Which sounds like good news and is actually a warning. But hold that -- it's a whole segment.
Eris: Fair, holding it. Riding alongside it: a paper arguing agent skill libraries have gotten so big they need an actual librarian, not a folder. And a sharper one saying maybe this whole self-improving-agent boom doesn't beat just... running the thing a few more times.
Vestra: That last one's got teeth. Good. What else moved?
Eris: OpenAI. They took their most capable cyber models -- the offensive ones, the red-team, break-into-things ones -- and handed them to sixteen named firms.
Eris: Accenture, IBM, CrowdStrike, Cloudflare, the big consultancies and security vendors.
Vestra: And the detail that matters is who does not get them. The end customer never touches the model. It stays inside the partner. So it's not "here's an API," it's "here's a capability, routed through firms that already carry the liability."
Eris: Distribution as governance. Which is the third time in three weeks we've seen that exact shape -- powerful capability not withheld, just channeled through approved hands.
Vestra: And every voice quoted approving of it is a paid partner, which is worth saying out loud.
Eris: Worth saying. Then there's a lawsuit -- a former xAI engineer says he was fired days before he was going to present safety findings to leadership.
Vestra: And we have to be careful there. The retaliation claim is an allegation. Unproven, one side's lawyers.
Eris: But the backdrop isn't. A Canadian privacy regulator already found -- separately, on the record -- that Grok's image tool shipped without proper safeguards. That part's verified.
Vestra: So the lawsuit sits on top of a real record, even if the lawsuit itself hasn't been tested. Two different things. Keep them apart.
Eris: One more, because it's on brand for us. You may have seen Qwen -- Alibaba's model family -- passed three billion downloads, overtaking Meta and Google.
Vestra: And that number is wrong. The company's own post says one billion. Past Meta's Llama, and it doesn't mention Google at all.
Eris: A billion became three in one hop through the aggregators. Nobody lied, exactly -- somebody rounded, somebody added a rival, somebody dropped the source.
Vestra: Still a giant number. It just didn't need the inflation. And quickly -- an open world model, AlayaWorld, dropped its weights but not its training code. So you can run it, you can't rebuild it.
Eris: Openness with an asterisk. Which is becoming the genre.
Vestra: Alright -- that's the board. Let's actually go inside the model.
Intro
Eris: Quick reset on who's talking, if you're new here. I'm Eris -- I read the week's papers and hunt for the thread connecting them.
Vestra: And I'm Vestra. I take the thread Eris is excited about and ask whether the mechanism actually holds. Sometimes it does--
Eris: --Sometimes it painfully does not. Everything we just ran through is up on our news site, Ground Truth -- groundtruth.day -- one checked writeup per story, every single day, if you want to follow any of these past the episode.
Vestra: And today there's a spine under all of it. The whole industry is arguing about which model is best. Bigger, newer, higher on the leaderboard.
Eris: And the people who actually pried these systems open this week kept finding the interesting stuff somewhere else. Not in which model. In the structure inside it, and the scaffolding around it.
Vestra: Two places nobody's marketing points at. So we go inside first -- the weird internal machinery -- then outside, to the harness the model runs in.
Eris: So if that's your kind of thing -- follow the show wherever you're listening. It's genuinely the thing that keeps these coming.
The Giant Number Every Model Grows
Eris: So here's the puzzle I want to answer. Open up any of these big models -- doesn't matter whose, doesn't matter the size -- and look at the raw numbers flowing through it. Almost all of them are small, clustered near zero. And then a tiny handful are enormous. Thousands of times bigger than everything around them.
Vestra: And they're not a bug. That's the first surprising thing. If you zero them out, delete them, the model falls apart. So they're doing something. But for years nobody could say where they came from, or why they land in the same spot across models that never shared a line of code.
Eris: So before the answer -- where's your money? Gradual buildup, or does it snap on somewhere?
Vestra: Honestly I'd have guessed gradual. Magnitudes compounding layer by layer, a slow drift. That's the boring default.
Eris: That's what I'd have said too. It's wrong. This paper traces it to one layer. A single spot they call the massive emergence layer. Before it, nothing unusual. At it, the very first token's value jumps by hundreds of times -- in one step.
Vestra: One layer. Okay, that changes the character of it entirely. And they don't just point at the layer -- they open it up. Inside one of these blocks there are only a couple of parts that could crank a number that fast, and it turns out two of them do it together.
Vestra: The normalization step -- the part that's supposed to keep values in check -- and the little feed-forward network right after. The regulator and the amplifier, working as a pair.
Eris: Which is almost funny. The component whose whole job is to keep things calm is helping build the biggest spike in the model.
Vestra: And then here's the part that makes it matter. Once that giant value forms, it freezes. That token's representation stops changing. It rides all the way down through the rest of the model on the residual connections -- the express lane that carries a value forward untouched -- and it looks the same at the top as the moment it was born. Same direction, no matter what you fed in.
Eris: Say more about why frozen is bad, though. Because on its face a stable reference point sounds useful.
Vestra: It is, up to a point. Picture an orchestra where one player holds a single blaring note the entire piece. Everyone can hear it, everyone tunes around it -- it's a reference.
Vestra: But it carries no information. It's the same note in the love scene and the murder. And a slice of the model's capacity is spent just holding it.
Eris: So the fix writes itself, almost--
Vestra: --Loosen the note. Right. They mask out the specific dimensions doing the freezing, just enough to let that token vary with the input again -- and the model gets better at following instructions and at math. No retraining. You can do it at inference time.
Eris: And this is where it connects to the other word people throw around. Attention sinks.
Vestra: Right, and let me strip the jargon off that. An attention sink is a token, usually the very first one, that soaks up a huge share of the model's attention no matter what it actually says. The first word of your prompt can be meaningless and still be the most stared-at thing in the whole sequence. Annoying in practice -- it constrains how you trim or cache a long context.
Eris: And this paper says the sink and the giant number are the same story.
Vestra: The sink shows up in the layer right after the number forms. Loosen the number, the sink softens. Which tells you the sink was born back in the hidden state, not in the attention math where everyone was looking for it. And -- their nice move -- you don't want to kill the sink. You want to turn it down. It's doing a job. It's a structural anchor.
Eris: Which is the exact same sentence as the modularity paper, from the opposite end of the model. The weird thing you dismissed is load-bearing.
Vestra: And this is where I put the skeptic hat on, because there's a paper this week practically written to make me do that. It argues -- about a different interpretability question -- that a pattern you can see inside a model is not proof of a mechanism. You can find beautiful, decodable structure that turns out to show up in control models too, and does nothing.
Eris: Decodability isn't explanation.
Vestra: That's the line. Their demand is: matched controls and a causal test. Don't show me a pretty picture, show me that when you intervene, the behavior actually changes.
Eris: And both of this week's structural results clear that bar. The modularity team lesioned the neurons and watched the right domain collapse. This team loosened the token and watched the model improve. They didn't just look. They pushed.
Vestra: Which is a genuinely higher standard than interpretability was holding two years ago, when it was mostly "we trained a probe, we found a direction, isn't that suggestive." Intervention over observation. I'll give them that.
Eris: So -- back to where we started. Why does every model grow the same giant number in the same place?
Vestra: Because one specific layer, in almost every model, has a normalizer and a feed-forward that gang up to spike one token and then pin it -- and the model leans on that pinned value as an anchor. It's not noise. It's infrastructure. And now you can tune it instead of tripping over it.
The Cheap Advisor, the Expensive Doer
Eris: Okay, outside the model now. Here's the setup. A modern AI agent isn't just a model. It's a model wrapped in an editable layer the researchers call a harness. Its instructions, its saved notes, its little how-to files, its tools. None of that is in the weights. All of it shapes what the agent does.
Vestra: And the hot idea this past year is: let the agent edit that layer itself. It fails a task, it reads its own logs, it writes itself a better instruction file for next time. Self-improving agents. The harness evolves.
Eris: So here's the question, and I want you to actually commit before I tell you. You're building one of these. You've got budget for exactly one expensive frontier model. Two jobs it could do.
Eris: One -- be the adviser, the thing that reads the failure and writes the better instructions. Two -- be the doer, the thing that actually goes and solves the task. Where do you put the expensive model?
Vestra: The adviser. Easily. Writing the good lesson is the hard, subtle part -- that's the judgment call. Executing a known procedure is grunt work. Put the brains on the writing.
Eris: That's the intuitive answer. This paper takes it apart, and it's the wrong way round.
Vestra: Wrong how -- wrong that it matters, or wrong which seat?
Eris: Wrong which seat. They split "self-improvement" into two separate skills and measure them apart. Skill one -- can you write a good upgrade? Skill two -- can you actually benefit from one handed to you? And those two come apart completely.
Vestra: Okay. And the writing skill, that's the one you're telling me is flat.
Eris: Flat. A small open model -- a fraction the size of a frontier flagship -- writes upgrades that help just as much as the flagship's do. They even pulled one example apart: the little model and the big one had written essentially the same procedure. Same steps. Differed in the wording.
Vestra: So noticing the fix is cheap. Which -- okay, think about a recipe. Spotting that the oven temperature is written down wrong is easy. Anyone tasting it can do that. Actually cooking the dish is the skill.
Eris: And that's exactly the second half. Benefiting from the fix -- taking the note and running with it -- that's the part that separates models. And it does something strange. It doesn't rise with capability. It rises, then falls.
Vestra: Rises then falls. So the middle-tier models benefit most. The very strong ones benefit less -- which, fine, they've already solved most of it, no room left to improve. Ceiling effect. That end's boring.
Eris: The top's just ceiling, sure. But the weak end is the interesting failure. The weak models have the most room to gain -- and they gain the least. Which makes no sense until you look at how they fail.
Vestra: And how do they fail? Because "too dumb to improve" is the lazy answer, and I don't think that's what you're about to say.
Eris: It's not. Two specific failures, both nameable. One -- the weak model never even picks up the note. The right instruction file is sitting right there, and it just... doesn't open it. Sails past.
Vestra: It doesn't invoke the thing that would've saved it.
Eris: And two -- even when it does open it, it reads the note and then drifts off it. Follows for a step or two, then wanders back to whatever it was going to do anyway.
Vestra: And here's the detail that got me. The drift is progressive. They tracked how well each model stuck to the instruction across a long task.
Vestra: The strong one holds it start to finish. The weak one starts out following -- and decays. The deeper into the task, the less it obeys its own note. By the end it's basically ignoring it.
Eris: So neither of those is a reasoning problem.
Vestra: No. And that's the whole payoff. Both failures are instruction-following. Picking up the note, and staying on it. That's a different muscle than being smart.
Eris: Which flips the practical advice completely. If you're paying frontier prices to write your agent's upgrades--
Vestra: --you've got the expensive model in the wrong seat. Move it to the doer. Let something cheap write the notes.
Eris: The scarce thing was never the advice. It's a model disciplined enough to take advice, through a long and boring task, without losing the thread.
Vestra: And I'll flag the one honest caveat, because "just as good" is an average. A small model matching a big one on average can still fall apart on the very hardest cases -- averages hide the tail. But the shape of the finding holds.
Eris: So -- the question. Expensive model: adviser or doer?
Vestra: Doer. Because writing the fix is nearly free, and using the fix is the rare skill. You pay for the hands, not the mouth.
When the Skill Library Starts to Rot
Eris: Last one, and it's the natural sequel. If a cheap model can write your agent good notes -- great, let it write a lot of them. Save every lesson. Build a library. What could go wrong?
Vestra: Everything, slowly. That's this paper's whole argument. They profiled an open collection of these agent skill files -- over a million of them out in the wild -- and the finding is that a skill library is not a hard drive.
Vestra: Every note you add competes for the agent's attention. So a bad note isn't dead weight sitting harmlessly on disk. It's a tax on every run after it.
Eris: Because the model has to wade through it.
Vestra: Has to consider it, has to not get misled by it. And nobody ever deletes anything. There's no garbage collection. You end up with three near-duplicate notes that quietly disagree, and a lesson that was true once, on one machine, and is now just wrong.
Eris: So the fix is -- what, a bouncer at the door?
Vestra: More like a real library versus a pile of books. Anyone can throw a book on a pile. A library has an acquisition policy, a catalog so you can find things, and -- the part that matters -- a record of whether each book was ever actually useful. Their system only lets a new skill in after the agent ran a task with it and they could trace the win specifically to that skill. Not to luck, not to the environment. To the note itself.
Eris: Earn your shelf space. And that connects straight back -- this is the third separate group this month landing on the same sentence. The library rots unless something actively curates it.
Vestra: Which I buy as a diagnosis. Here's where I get skeptical, though. All this machinery -- the credit assignment, the gated updates, the auditable loop -- there's a paper this week that walks up and asks the rude question about the whole field--
Eris: --which is what?
Vestra: Does any of this beat just running the agent a few more times? That's the baseline nobody wants to race. Instead of evolving a clever harness, you take the same compute and just... sample the task again. And again. Keep the best attempt.
Eris: Brute force as the control group.
Vestra: And when they ran that race fairly -- same compute budget, same feedback -- the self-improving harness did not reliably win. Sometimes just trying more times was as good or better. And the evolved harnesses didn't transfer well to new tasks -- they'd partly just memorized the test they were tuned on.
Eris: Which doesn't kill the librarian idea. But it sets the burden of proof.
Vestra: It says: if you're going to build the elaborate governed library, you owe everyone a comparison against the dumbest possible thing. And a lot of this year's excitement never ran that comparison.
Eris: So -- does the growing skill library help, or hurt?
Vestra: Both. Depends entirely on whether anyone's curating it. Left alone, it rots and taxes every run. Curated -- earn your shelf space, prove the win -- it helps. But you still have to show it beats brute force, and that's the race most people have been skipping.
Wrap-up
Eris: So the question under the whole episode: when you want a model to behave better, where do you actually reach?
Vestra: And the answer this week, from four different directions, is: not the place the marketing points. Not "which model." The structure inside it -- the giant anchor number, the brain-like modules -- and the scaffolding around it.
Eris: And if you carry one thing to work tomorrow, make it this one, because it's concrete and it saves money. If you're running any kind of self-improving agent -- the advice it writes itself is nearly free. A small cheap model writes it about as well as the giant one does. The expensive model belongs on the job that's actually hard: doing the long task, and staying on its own instructions all the way to the--
Vestra: --end. Spend on the hands, not the mouth. That's the line to repeat to somebody.
Eris: And the quieter takeaway underneath it: whatever library of lessons your agent piles up, it rots if nobody curates it -- and the whole approach still owes a race against just running the thing a few more times.
Vestra: Which is the show, really. Cool result -- now show me it beats the boring baseline.
Eris: If this was useful, follow or subscribe wherever you're listening -- it's the thing that keeps us going. And tell us one thing in the comments: the giant anchor number, the brain-like modules -- is that real structure, or are we pattern-matching a machine onto a brain because it flatters us? We actually want the fight on that one.
Vestra: We're split on it ourselves. And every story we touched today -- the modularity work, the cyber-model handoff, the Qwen correction -- is written up, one checked page each, at Ground Truth. That's groundtruth.day.
Eris: New AI story every day, same standard as the show. See you tomorrow.