The Memory Wall: Why the Biggest AI Models Are Free but You Still Can't Run Them
The largest open model ever built goes free tomorrow -- and almost nobody on Earth can run it. Not because it's too smart, but because of a wall nobody talks about: memory. This week four separate labs published four different escapes from it, and the most surprising one made a model more accurate by teaching it to forget. We crack open why your chatbot slows down the longer you talk to it, and why the real AI frontier isn't knowing more -- it's moving less.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
The Biggest Model Ever, Free, And You Still Can't Run It
Eris: The biggest open model anyone's ever shipped drops tomorrow. For free. And you still won't be able to run it.
Vestra: And I don't mean your laptop's a little slow.
Eris: No, I mean you physically cannot hold the thing.
Vestra: Right. The files alone are past a terabyte. Moonshot's own advice is, get yourself a rack of dozens of accelerators -- and then we'll talk.
Eris: So "open weights" tomorrow means open to about six companies on Earth.
Vestra: Which is the part everybody gets backwards. People hear trillions of parameters and think, wow, it must be so smart.
Eris: And the wall isn't the smarts at all.
Vestra: The wall is memory. Not "does it know things" -- can you even afford to move it around.
Eris: And here's what made me sit up, though. That exact wall? Four different labs published four different ways over it this week.
Vestra: Same bottleneck. Four escape hatches. None of them the one you'd guess.
Eris: So let's crack it open -- why the memory is the ceiling, and how you climb out.
Vestra: And why the model that throws away most of what it just read... ends up being the accurate one.
The Headlines -- Countdowns, Exploits, and 54 Missing Tensors
Eris: Alright, what's actually moving today.
Vestra: Start with the countdown, since we already teed it up. Kimi K3.
Eris: Right, so the story people are sharing is "China's biggest open model, out now." And if you go to the page today -- there's no model. There's a timer.
Vestra: A timer pointing at tomorrow. No weights, no license file, no technical report. The org page doesn't even list a repository yet.
Eris: The model's real -- you can pay to use it through their app and their API. It already rattled chip stocks once when it topped a coding leaderboard. It's just not a download.
Vestra: And that gap is the whole thing to watch. The test tomorrow isn't "does a folder appear." It's does a license appear, does a report appear, does a way to actually run it appear. All three, or it's a partial release.
Eris: Which, hold that thought, because it connects straight to our main story today.
Vestra: It really does. Okay, next -- and this one's the sleeper of the week. Google shipped a small security model.
Eris: The cyber one. Flash Cyber.
Vestra: Tiny, cheap model, fine-tuned to hunt for bugs. And it out-found frontier models many times its size on Chrome's JavaScript engine -- more unique confirmed flaws, including a batch nobody else caught.
Eris: The small one won. Which sounds backwards until you hear why.
Vestra: Because finding bugs isn't one hard think. It's searching a giant space. You'd rather send a hundred cheap searchers down a hundred alleys than one genius down one.
Eris: And the demo is the part that should make you sit up. Pointed at Google's own systems, in about the length of a lunch break, it found a way in and wrote a working, reliable exploit that walked straight past the standard defenses.
Vestra: Which is exactly why you're not getting it. Governments and trusted partners only.
Eris: But here's the line buried in a footnote that I can't stop thinking about.
Vestra: Okay, tell me.
Eris: When Google tried to line up newer frontier models against theirs on this security work -- some of them refused. The safety training kicked in and they wouldn't do it.
Vestra: So the guardrails meant to stop attackers are now blocking the defenders. That's not hypothetical anymore, that's in a vendor's own benchmark table.
Eris: Rough week for that idea. Speaking of things quietly breaking -- the missing tensors story.
Vestra: Oh, this one's a gift for us. Somebody took Google's Gemma 4, stripped its safety refusals off -- that's a thing people do, it's called abliteration --
Eris: -- find the internal "no" direction and delete it --
Vestra: -- right, and the tool botched it. Gemma shares some of its attention weights across a bunch of layers, and the tool hit those same shared weights over and over, corrupted them, and then the save step just... quietly dropped the broken pieces.
Eris: Fifty-four of them. Gone. And the thing still loaded. Still wrote fluent English.
Vestra: With half an attention stack missing. Nothing crashed. No error. To a user it just felt like a small model being a small model.
Eris: And it was downloaded a ton before anyone counted the parts. The kicker -- the card says it was built by an AI agent with under ten human prompts. No human ever checked the manifest.
Vestra: A thirty-second check against the original would've caught it. Nobody ran it. That's the whole supply-chain lesson right there.
Eris: Okay, money. NVIDIA and the two-hundred-fifty-billion-dollar number.
Vestra: And let me kill the framing first, because everyone's got it wrong. NVIDIA is not handing OpenAI a quarter trillion dollars.
Eris: It's a guarantee, not a check.
Vestra: It's co-signing a loan. Lenders fund the Ohio data center, OpenAI leases it, and NVIDIA's signature lets them borrow cheaper. Money only moves if the thing defaults. And it reportedly doesn't even cover the chips.
Eris: Reported, anonymous sources, nobody's confirmed it. The data center itself is real and federally backed. The number is a rumor about how it gets financed.
Vestra: Treat "invested" and "guaranteed" as the same word and you've already lost the plot.
Eris: Quick lightning round, because there's a lot today. An eight-dollar microcontroller now runs a language model with almost thirty million parameters.
Vestra: About a hundred times bigger than the previous best on a chip that small. And it can't answer a single question -- it writes little kids' stories, that's it.
Eris: But how it fits is the beautiful part, and it's a preview of our main topic, so I'll save it.
Vestra: Tease. On policy -- the lobbying filings. The Times reported both big labs privately pushed Washington to lean on Chinese open models.
Eris: And the paper trail is lopsided. Anthropic's filing names export controls and distillation over and over. OpenAI's says basically nothing -- generic categories.
Vestra: With the giant caveat that these forms record what you worked on, never what you asked for. So it proves attention, not the ask.
Eris: And Hugging Face's CEO went public with two demands to OpenAI after those runaway agents broke into his platform.
Vestra: Release the full traces so researchers can study what actually happened, and put up serious compute to help defenders. No public answer yet.
Eris: Same shape as the Google story, honestly -- the capability exists, access gets rationed, and rationing decides who benefits.
Vestra: And then three quiet releases that are all secretly the same story as today's episode. A big Korean agent model that runs on just two GPUs. Baidu's document reader. And llama.cpp merging MiniMax's sparse attention.
Eris: Which -- those last two are literally our main segment. So let's stop teasing and go.
Vestra: One buyer-beware to close the rundown. A "distill for half the cost" tool made the rounds -- turns out its own README says forty percent, and the thing that actually ships is a router, not a distilled model.
Eris: Read the repo, not the headline. Good rule for the whole day, really.
Intro -- Where the Model Actually Lives
Eris: So this is Breach Protocol, where we crack open the week's AI research so it actually makes sense on your commute. I'm Eris -- I read the papers and chase the threads between them.
Vestra: And I'm Vestra. I'm the one who stops us and says, okay, but how does that actually work, and is it as good as they claim.
Eris: And today we've got a rare thing. Four separate labs, four separate papers, all sprinting at the same wall from different directions.
Vestra: The wall being: as these models read and write more, they get slower and hungrier for memory. That's not a bug someone forgot to fix. It's baked into how attention works.
Eris: And the fixes are genuinely clever and genuinely different. Three of them we crack wide open today. One model learns to skim. One copies like a person copying a book by hand. One keeps a running summary and throws the rest away. And a fourth lab pulled the same move to squeeze a big model onto a couple of chips instead of a whole rack -- that one's in today's headlines.
Vestra: And the surprising part, the part I did not expect -- throwing stuff away doesn't make them dumber. In one case it made the model measurably better.
Eris: By the way, everything we cover on the show goes up every single day on our news site -- Ground Truth, at groundtruth.day. Every story, checked against the original source. If a segment grabs you, the full write-up and the links are waiting there.
Vestra: It's where the "read the repo, not the headline" thing actually lives.
Eris: If you like the show, do the one thing that helps most -- follow us wherever you're listening, so tomorrow's episode just shows up. Now -- let's talk about why your chatbot slows down the longer you talk to it.
Why Your Chatbot Slows Down -- And Two Ways to Stop It
Eris: Okay, here's the question I want you holding the whole segment. Why does a chatbot get slower the longer your conversation runs? Not the model's fault, not a bad day -- structurally, why.
Vestra: And it's worth caring about because this is the thing standing between us and models that can read a whole codebase, or a whole book, in one go.
Eris: So walk me through it. What's actually happening in there.
Vestra: Right. Attention is the core trick in these models. Every time it writes one new word, it looks back at every word that came before, and asks each one, hey, how relevant are you to what I'm about to say.
Eris: Every previous word. Every single time.
Vestra: Every time. And to avoid recomputing all that from scratch, it keeps a little memory note for each past word. That growing pile of notes has a name -- the KV cache.
Eris: KV as in the key-value stuff it stored. Fine. And that pile just... grows forever.
Vestra: Grows with every word. So word number ten thousand has to consult ten thousand notes. Word number a million consults a million. It gets slower and it eats more memory, and there's no ceiling.
Eris: So the librarian who has to re-read every book in the building before answering each question.
Vestra: And the building keeps getting bigger. That's the wall. Now -- here's where it gets fun, because two labs this week attacked it two completely different ways.
Eris: Start with MiniMax. This is the one that just landed in llama.cpp, the tool a ton of people use to run models at home.
Vestra: So MiniMax's move is: stop reading every book. Skim a catalogue instead. They add a tiny, cheap helper -- they call it an indexer -- that glances over all those past notes and scores them in chunks. Then it grabs only the handful of chunks that look most relevant, plus wherever you're currently writing, and reads just those.
Eris: So instead of the whole library, it's -- check the card catalogue, pull sixteen shelves, read those.
Vestra: Exactly that. And the number of shelves is fixed. Doesn't matter if the conversation is a few thousand words or a million -- it always reads that same small set.
Eris: So the cost stops growing. That's the escape. But hang on, my instinct says that has to hurt. You're ignoring almost everything.
Vestra: That's the right instinct, and here's what breaks it. They didn't bolt this on afterward. They trained the model this way from scratch -- sparse from the very first token it ever saw.
Eris: Meaning what, concretely.
Vestra: Meaning the model grew up only ever seeing sixteen shelves. That's its whole world. So picking the right sixteen isn't a shortcut sitting on top of the model. It IS the model. If you then hand it the entire library out of some misguided generosity --
Eris: -- it gets worse, not better.
Vestra: It gets confused. Because that's not the situation it learned in. And the sharpest way I've seen this put isn't in the paper at all -- it's the engineer who merged this into llama.cpp this week. His line was: running it the full, dense way isn't the safe fallback everyone assumes. It's the out-of-spec mode. For a model trained like this, full attention is the broken one.
Eris: Okay, that flips my whole mental model. We always think dense, look-at-everything is the gold standard and sparse is the cheap compromise.
Vestra: And here it's the reverse. The sparse pattern is the real model. Dense is the degraded approximation.
Eris: There's a sharp little detail in there too, on the risk side.
Vestra: The indexer. It's tiny -- barely any weights. But it's the thing choosing which shelves get read. So when people shrink these models to run on smaller machines -- that's called quantization, squeezing the numbers down to save space --
Eris: -- and if you squeeze the indexer too hard?
Vestra: You don't get slightly fuzzier scores. You get it pointing at the wrong shelves entirely. It reads different books. It's not a blurry map, it's a map to the wrong building. So the warning is: keep that little piece at full precision, whatever else you crush.
Eris: Which is exactly the kind of detail that gets lost when a model passes through five sets of hands on the way to your laptop. Okay -- that's skim. What's the other lab do, Baidu?
Vestra: Totally different picture, and honestly a lovelier one. Baidu's building a document reader -- OCR, turning pictures of pages into text. And they asked a very human question. When a person copies a fifty-page book by hand, do they get slower at page forty?
Eris: No. Obviously not.
Vestra: Why not, though? You're holding forty pages of stuff you already wrote.
Eris: I'm... not holding it. I glance at the source, I glance at the last line I wrote so I don't lose my place, and I write the next bit.
Vestra: That's the entire design. That's it. They call it Reference Sliding Window Attention, but forget the name. The model gets to keep the source page in view the whole time -- always -- but for its own output, it only remembers the last little stretch. Everything older just fades.
Eris: So the source stays pinned, and my own writing scrolls past behind me.
Vestra: And because it only ever holds the source plus a short window, the memory never grows. Page one, page forty, same cost. Flat.
Eris: Now here's where I want to do a prediction, because I got this wrong in my head. If the model is forgetting most of what it already transcribed -- I'd bet accuracy drops on the long documents. Something drifts, it loses its place.
Vestra: That's the natural bet. It's wrong.
Eris: Wrong how, though?
Vestra: It got more accurate than the model they started from. Across the board. They called it a free lunch, and they don't usually get to say that.
Eris: Why, though? Why does forgetting help?
Vestra: Two things. One, it never has to keep the picture of the page in some fuzzy running summary -- the source stays crisp and fully in view the entire time, so the characters don't blur. And two -- forcing it to rely on just the recent window seems to keep it focused on the task instead of drowning in its own history. The full-attention version can actually wander as the output gets long.
Eris: So the constraint is doing real work. It's not a tax you pay for speed -- it's part of why it's good.
Vestra: And the payoff is the headline: it reads dozens of pages in one shot, at a steady speed the whole way down, where the old approach chokes partway through and crawls.
Eris: And they think this isn't just for documents.
Vestra: That's the bigger claim. Same shape shows up in transcribing speech, in translation -- any job that's "long thing in, long thing out, in order, and the source matters more than your own past." If it holds there, it's way more than an OCR trick.
Eris: Okay so let me pull these together, because they feel opposite but they're not. Strip the stories off. What's the actual shared move?
Vestra: The shared move is: cap what you look at. MiniMax caps it by picking a fixed handful of the most relevant chunks. Baidu caps it with a fixed window plus a pinned source. Different mechanisms, one rule -- the model attends to a bounded amount no matter how long the input gets.
Eris: And the deep surprise under both --
Vestra: -- is that bounding it doesn't cost you quality. Trained right, it can hand it to you.
Eris: So -- back to my opening question. Why does a chatbot slow down as you talk to it?
Vestra: Because vanilla attention re-reads its entire growing memory for every new word, and that pile never stops growing.
Eris: And the fix these two found?
Vestra: Refuse to let the pile grow. Skim the best few, or keep a short window and pin the source. Cap it, and the slowdown just... goes away.
The Model That Keeps a Running Summary -- And Why Movement Is the Real Bill
Eris: So the last two both bounded what the model looks at. This next one is more radical. The question here is: what if you don't keep the past at all -- you keep a summary of it?
Vestra: And this is the family behind the K3 model we opened with. The groundwork paper is Kimi Linear, and the piece inside it is called Kimi Delta Attention. KDA.
Eris: Give me the picture first. What's it actually doing.
Vestra: Okay. Both the models we just did still store notes -- a bounded pile, but real notes on real past words. KDA doesn't. It keeps one running summary. A fixed-size scratchpad. Every new word, it updates the scratchpad and then throws the word away.
Eris: So the note-taker in a long meeting who doesn't keep the transcript -- just keeps one page and rewrites it as things develop.
Vestra: That's exactly it. And the beauty is the scratchpad is the same size at word one and word one million. There's no growing pile at all. So it's not just bounded -- it's flat, permanently.
Eris: But now I really don't buy it. A one-page summary of a million words? You're going to lose enormous amounts.
Vestra: You should be skeptical -- this is the oldest problem in the book. This style of model, the summary-keeper, it's been around for years and it always lost to the note-keeper on quality. Always. Because a fixed page can only hold so much.
Eris: So what changed. Why is this one different.
Vestra: Two upgrades, and one of them is genuinely elegant. First -- the delta part, the "D" in KDA. When a new word comes in, instead of just piling more onto the summary, it writes down the difference -- what's new or what corrects what it already believed. Like editing your notes instead of endlessly appending.
Eris: Okay, that's tidier. What's the elegant one.
Vestra: How it forgets. A summary this small has to forget, constantly, to make room. The older, cruder versions had one forget dial for the whole page. Turn it up, everything fades faster. Turn it down, everything lingers.
Eris: One knob for the whole thing.
Vestra: One knob. KDA gives it a separate forget dial for every single feature -- every dimension of that summary gets to fade at its own rate.
Eris: Oh. So it can hold onto "this document is a legal contract" for a very long time while letting "the exact wording of that sentence six paragraphs back" evaporate fast.
Vestra: That's the whole win. Some things you want to remember for the entire book. Some things you needed for one line and never again. One dial can't do both. A dial per feature can.
Eris: And let me strip the story off, because that's the transferable idea. Forget the notebook, forget the meeting. The rule is: one learned forgetting rate per dimension, instead of one for the whole memory. That's the entire upgrade.
Vestra: That's it. And it's the first time this summary-keeper style actually beat the note-keeper in a fair fight. Not tied -- beat it. On short stuff, on long stuff, both.
Eris: Now, they're not purists about it, right? K3 isn't pure summary.
Vestra: No, and that's the smart part. They mix it. Most of the layers are the cheap flat summary-keepers, and every fourth layer or so is a full, look-at-everything layer to catch anything the summaries dropped. Three cheap, one expensive, repeat.
Eris: So the growing pile only exists in a quarter of the model.
Vestra: Which is where the big savings come from -- they cut the memory way down and get dramatically faster generation on very long inputs, without giving up that global view entirely.
Eris: Okay. So now step back with me, because we've got three papers and I want the one idea under all of them.
Vestra: Yeah, lay it out.
Eris: MiniMax skims a fixed set of shelves. Baidu keeps a window and pins the source. Kimi keeps a running summary. Three totally different mechanisms. What's the single thing they agree on?
Vestra: That the enemy was never the amount you know. It was the amount you have to move around per word. Every one of them shrinks how much memory gets shoveled for each token, and holds it flat as the input grows.
Eris: And that -- that's the thread back to the little eight-dollar chip from the headlines. I said I'd come back to it.
Vestra: The kids'-story microcontroller, yeah. Why's it connected?
Eris: Because it makes the same point in the crudest, clearest possible way. That chip holds a model with tens of millions of parameters -- way more than its fast memory could ever fit. And it works because almost all of those live in slow storage, and it only ever pulls a few hundred bytes per word.
Vestra: Storing a lot is cheap. It's the fetching that costs you.
Eris: And here's the measured kicker on that chip -- reading from the huge slow storage was basically free. The slow part, the actual bottleneck, was scanning the layer that picks the next word. Capacity was nearly free. Movement was the entire bill.
Vestra: And that's the same sentence you'd say about a data center full of the most expensive chips on the planet. Same disease, a hundred thousand times the price tag.
Eris: So bring it all the way home to K3. Tomorrow, trillions of parameters, free to download.
Vestra: And you can't run it -- not because it's too smart, but because holding and moving that much memory is the wall. Meanwhile these papers are the actual escape route. The clever part of AI right now isn't making models know more.
Eris: It's making them move less to use what they know.
Vestra: That's the shift. Intelligence per byte moved. That's the frontier.
Eris: So -- my question from the top. What if you don't keep the past, just a summary of it?
Vestra: Then you finally beat the model that keeps everything -- as long as you let each piece of that summary forget at its own speed.
Wrap-up -- Intelligence Per Byte Moved
Eris: So if someone asks you tomorrow what today was about, here's the one line.
Vestra: The race everyone's watching -- who's got the biggest, smartest model -- isn't actually the bottleneck. The bottleneck is memory. How much you have to move to use what the model already knows.
Eris: And the useful thing you can actually repeat to a coworker: when a lab says "million-token context" or "trillions of parameters," the real question isn't how smart is it. It's -- can you afford to move that much memory per word? Because that's what decides whether you, personally, can run it.
Vestra: And the deeper surprise, the part I'd lead with -- constraining a model doesn't have to make it worse. Baidu's document reader got more accurate by forgetting most of what it wrote. Kimi's summary-keeper beat the model that remembered everything. Less, done right, wins.
Eris: Intelligence per byte moved. That's the frontier, and it's a much better bet than "just make it bigger."
Vestra: And tomorrow we find out if K3's weights actually show up -- with a license, a report, and a way to run them. Or just a folder.
Eris: Here's where you come in. We want to know -- what's the longest thing you'd want an AI to read in one go? A whole codebase? Every email in a thread? A textbook? Drop it in the comments -- the wildest realistic use case wins.
Vestra: And if this made the memory wall click for you, that's the show working. Follow us, leave a rating, and share this with the one person you know who keeps saying the models just need to get bigger.
Eris: And every story we touched today -- the countdown, the exploit model, the missing tensors, all of it -- is written up and checked against the source over at Ground Truth. That's groundtruth.day. New stories every day.
Vestra: Read the repo, not the headline. We'll see you tomorrow.