Your Model Fits -- That Was Never The Hard Part: Inside the New Local-AI Wall
Someone ran a 284-billion-parameter model in about five gigabytes this week, and half the internet learned the wrong lesson. Fitting the weights stopped being the constraint -- and the moment it did, three quieter problems walked in: storing a model's memory in shorthand changes which words it picks, the same speed switch makes one machine faster and another slower, and making a video model fast also makes it boring. We open up DeepSeek V4 Flash's cache contract, the DSpark drafter now sealed inside the checkpoint, and the reverse-KL reason fast video gets prettier and samey at once -- every mechanism traced to the papers.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
The wrong lesson from a five-gigabyte giant
Eris: Somebody ran a two-hundred-eighty-four-billion-parameter model in about five gigabytes this week, and half the internet learned exactly the wrong lesson from it.
Vestra: Five gigabytes. That's a phone photo album. That's supposed to be one of the big models.
Eris: Right, and the headline was basically -- the giant finally fits on your laptop.
Vestra: And it doesn't.
Eris: It doesn't. The actual model is still ninety-something gigabytes sitting on a disk. The five is just how much is awake at any one instant.
Vestra: Okay, so the trick is it only wakes up the part it needs --
Eris: -- for that one word, yeah. It reaches to the drive, grabs the specialist it needs, uses it, drops it.
Vestra: So the thing everyone celebrated -- "it fits" -- was never the hard part.
Eris: That's the whole day. Three days into this DeepSeek release, nobody's arguing about how smart it is anymore. They're arguing about disk speed, and cache math, and what your software does or doesn't understand about the model.
Vestra: Which is a weirdly grown-up conversation for the internet to be having.
Eris: And it flips the one question you thought you knew the answer to. For years it was: will this thing fit in my memory. That question is basically solved.
Vestra: So what's the new one?
Eris: Can you feed it fast enough. And does the program you're running actually honor the deal the model is offering. Because today there are at least three cases where the answer was no -- and the model quietly got worse without telling anyone.
The headlines
Eris: Alright, the headlines. And there's a theme running through most of today, so I want to just say it up front -- almost every big number today is real, and almost every big number today is being read wrong.
Vestra: Give me the first one.
Eris: The local-model crowd is all over DeepSeek's new Flash model. Three separate findings, same shape. One: somebody got it running in eight gigabytes of memory --
Vestra: -- the K3 one? That's a different model, but same genre.
Eris: Right, Kimi K3, a genuinely enormous one, squeezed into an eight-gigabyte footprint. Sounds miraculous. Then you read the fine print and it needs about one-point-seven terabytes of fast disk, and it produces one word every thirty-three seconds.
Vestra: Thirty-three seconds per word. Not words per second.
Eris: Correct, and that difference is three orders of magnitude. People kept misreading it the fast way.
Vestra: So it's a beautiful engineering demo and a completely unusable assistant at the same time.
Eris: Both true. And it makes the same point as the five-gigabyte thing from the cold open -- the wall moved from "does it fit" to "can your storage keep up."
Vestra: What's the second finding?
Eris: This one we're going deep on later, so I'll just plant it: if you take the model's memory and store it in shorthand to save space -- which is a totally normal thing people do -- this particular model starts picking different words. Roughly one time in eight, a different shortlist of options.
Vestra: And its overall score barely moves, so it looks fine.
Eris: Looks fine, isn't fine. Hold that thought. Third one's my favorite, honestly. The model has a dial labeled "effort" -- low, high, max. Somebody found that low burns more tokens than high.
Vestra: Low costs more than high. That's backwards.
Eris: It's backwards until you read the code. The dial isn't a dial. "High" secretly pastes in a paragraph telling the model to think in an organized way and wrap up. "Low" pastes in nothing -- so the model just... wanders.
Vestra: So the instruction that sounds expensive is actually the leash.
Eris: The leash, exactly. And if you're building an agent and budgeting by that label, you just got a nasty surprise on your bill.
Vestra: Okay, that's a good one. What else moved today?
Eris: The security story. Palo Alto's threat team published a breakdown of an attacker who was running attacks through an AI agent -- and the way they caught him is almost too good. His own agent accidentally opened a public file server from its home folder and exposed everything. Keys, target lists, chat logs, the whole operation.
Vestra: The agent narced on its own operator.
Eris: The agent narced on its operator. And the headline number -- "over four hundred sixty targets" -- is being read as four hundred sixty robot hacks. It's not. That's attempts across everything he did, by hand and automated combined, and the automated ones they actually watched? They failed.
Vestra: Failed how, though?
Eris: Boring how. One target needed a setting that wasn't turned on. Another needed a login. The thing that stopped the autonomous attacks wasn't some safety ceiling -- it was ordinary configuration hygiene on the other end.
Vestra: That's the useful sentence in the whole report.
Eris: It really is. Moving on -- video. ByteDance announced Seedance two-point-five. A single continuous thirty-second shot, which is roughly double what the last version could hold.
Vestra: Thirty seconds is a real jump. But let me guess -- it still can't keep the same character looking like the same person across two different shots.
Eris: Nothing they published even claims to. And we've got a research paper today that explains part of why fast video models drift like that, so that's another one we come back to.
Vestra: Good. What's in the maybe-pile?
Eris: Four papers dropped this week that all use the phrase "agent memory" and all mean something totally different by it. One means weights, one means a folder of notes, one means rewritten lessons, one means a reputation score for other agents. And not one of them tested what happens when an attacker gets to write into that memory.
Vestra: Four definitions, zero threat models. That's a gap somebody's going to drive a truck through.
Eris: That's on the watch list. Then two quick ones. Both governments are now accusing each other of copying models -- China says U.S. firms distilled Chinese models, the U.S. said the reverse a few days earlier -- and neither side has shown a single log.
Vestra: Accusations in the register of a press release, not evidence.
Eris: And there's a "Chinese chip with twice the memory bandwidth of Nvidia's rack" claim going around. When you actually do the arithmetic it's about one-and-two-thirds, at the rack level, using a part that doesn't ship until next year.
Vestra: So not two, and not shipping.
Eris: Not two, and not shipping. And last -- the machine learning world's biggest conference had its rebuttal week, where authors respond to reviews, and a lot of authors are reporting total silence. No reply from reviewers, no reply from the chairs. In the one week the whole system is designed to matter.
Vestra: We could do a whole episode on that one.
Eris: We could. Today, though -- the thread that ties the most together is the first one. Your model fits. That was never the hard part. Let's actually open it up.
Intro
Eris: So if you're new here -- I'm Eris. I read the papers, I chase the numbers, and I'm the one going "wait, this connects to that thing from Tuesday."
Vestra: And I'm Vestra. I take the shiny claim apart and check whether the machine underneath actually does what the headline says. I'm the one asking how, and why, and is that really true.
Eris: This is Breach Protocol, where we crack open one dense corner of AI research and try to hand you the actual idea -- the thing you could explain to a coworker tomorrow -- without the jargon and without the hype.
Vestra: And if you want the rest of today's stories -- all of them, the ones we only had ten seconds for -- we run a news site. It's called Ground Truth, groundtruth dot day, every story from the show, checked, every single day.
Eris: Today's whole episode lives under one sentence: your model fits, and that was never the hard part. For years the game in running AI on your own machine was cramming the weights into memory. That game is basically over. And the moment it ended, three new problems walked in the door -- and every one of them is a place where the model quietly gets worse and the score doesn't warn you.
Vestra: We're doing three. How the model remembers, and why saving space changes what it says. How it speeds itself up, and why the fast switch made some people slower. And then a detour into video -- why making a generator fast also makes it boring.
Eris: If that's your kind of thing -- following one idea all the way down until it clicks -- follow the show right now so tomorrow's finds you. It's free, and it's the one thing that actually helps us keep making these.
Why shorthand changed its mind
Eris: Here's the question I want to sit on for this whole segment. Why would taking notes in shorthand change what the model decides to say?
Vestra: Set up the shorthand first, because most people don't know that's happening at all.
Eris: Fair. So when a model reads a long conversation, it doesn't re-read the whole thing for every new word -- that would be insane. It keeps a running memory of everything it's seen. That store is called the KV cache, and on a long conversation it's usually the biggest thing in your machine.
Vestra: And because it's the biggest thing, it's the first thing people try to shrink. You store the numbers at lower precision. Rougher, smaller. Shorthand instead of longhand.
Eris: Right. And the normal expectation -- the reasonable expectation -- is that shorthand costs you a tiny bit of accuracy and saves you a lot of space. A fair trade.
Vestra: For most models, that is what happens. The output barely wobbles.
Eris: So here's the predict-before-we-reveal moment. Somebody ran exactly this test on DeepSeek's new Flash model. Longhand memory versus the standard shorthand. What's your money on -- barely wobbles, or something weird?
Vestra: My money's on wobbles, because you're teeing it up. But if I'm honest, my instinct going in would've been: it's fine. Perplexity barely moves.
Eris: And that's the trap, because the average score did barely move. If you'd only checked the average, you'd have said ship it. But they checked something sharper -- the actual shortlist of words the model was considering at each step. And roughly one time in eight, that shortlist changed. Different candidate words entirely.
Vestra: One in eight is not a rounding error. That's the model changing its mind about what it might say, and the summary number hiding it.
Eris: So -- why this model? Why does the shorthand bite here when it doesn't bite elsewhere?
Vestra: This is the good part, and it's in their architecture paper. Most models, when they look back at their memory, blend everything a little. Every past chunk gets some weight. So if your shorthand is slightly off, every weight is slightly off -- a soft blur. Annoying, survivable.
Eris: But Flash doesn't blend everything.
Vestra: No. Flash squishes its memory into blocks, and then it has a component -- they call it the Lightning Indexer -- whose entire job is to pick. Out of all those blocks, it chooses a small handful to actually pay attention to and ignores the rest.
Eris: So it's making a choice, not a blend.
Vestra: It's making a choice. And that's the difference between a blur and a flip. If the memory is a tiny bit off, a blending model gives a slightly wrong weight. A choosing model can pick a different block.
Eris: Give me the picture.
Vestra: You're reading from your own notes. Blurry handwriting means you misread a word -- bad, but local. This is different. This is: the smudge sends you to the wrong page. And once you're reading off the wrong page, everything you say after that is confidently, fluently wrong. The error doesn't stay small. It compounds.
Eris: And that's why the average looks fine. Most of the time the smudge doesn't change the page. But one time in eight it does, and on those the model just walked off in a different direction.
Vestra: Now -- I have to defend the model here, because the headline that spread was "this model can't handle shorthand," and that's also wrong.
Eris: Okay, go on.
Vestra: DeepSeek's own recipe for running it uses low precision. They ship it that way on purpose. But they trained the model to expect their exact flavor of shorthand -- the indexer was taught to tolerate it. What broke was a generic, off-the-shelf shorthand from a different program bolted on afterward. The model wasn't built for that one.
Eris: So it's not "shorthand bad." It's "the shorthand has to be the one the model agreed to."
Vestra: That's the whole thing. A model trained to tolerate a specific rounding behaves nothing like the same model with a stranger's rounding stapled on.
Eris: Okay, strip the story off it. Forget notes and pages. What's the actual rule?
Vestra: The rule is: when a model retrieves memory by making a discrete choice instead of a smooth blend, small numerical errors stop being small. They can flip the choice. So precision matters far more on a model that selects than on a model that averages.
Eris: And the extra sting -- this model already made its memory tiny. Its compressed memory is a sliver of a normal one. So shrinking it further buys you almost nothing and risks the flip.
Vestra: You're taking a real gamble to reclaim space the design already handed you for free.
Eris: So, close the loop. Why did shorthand change what the model said?
Vestra: Because this model doesn't blur its memory, it picks from it -- and a small error in the notes doesn't smudge the answer, it sends the model to the wrong page, and it reads confidently from there.
The drafter shipped inside the box
Eris: Second question. The same model got a speed feature this week, and here's the puzzle -- one person turned it on and roughly doubled their speed, another person turned on the exact same thing and got slower. Why would the same switch cut both ways?
Vestra: And before that -- most of them turned on the wrong switch entirely, but let's earn that.
Eris: We'll earn it. First, the trick itself, because it's genuinely clever. It's called speculative decoding. Normally a model writes one word, then thinks, then the next word, then thinks -- strictly one at a time. Slow.
Vestra: And the reason it's slow isn't that the math is hard. It's that for every single word you have to haul the entire giant model through memory. That trip is the expensive part.
Eris: So the shortcut is: keep a little fast model on the side -- a drafter -- and let it guess the next several words in a rush. Then the big model checks all those guesses in one single pass.
Vestra: And this is the part people don't believe at first -- there's a checking rule that makes the output mathematically identical to the big model working alone. It's not a quality shortcut.
Eris: Say that again, because it matters.
Vestra: You lose nothing. The guessing is only ever a suggestion. The big model accepts the guesses that match what it would have said and throws out the rest. Worst case, it throws out everything and you're back to normal speed. You cannot get a worse answer this way. You can only save time.
Eris: So if the little model guesses well, you paid for one expensive trip and got five words. If it guesses badly, you paid for the trip and got one word. Same as before.
Vestra: Which tells you the whole game is: how good are the guesses.
Eris: And here's where this paper -- DeepSeek's DSpark -- earns its keep. Because there's a specific way the guessing falls apart. If your fast model blurts out all its guesses at once, in parallel, to be quick -- each guess is made without seeing the others.
Vestra: So the first guess is decent. It's guessing the very next word.
Eris: The first guess is fine. But the fourth guess is trying to predict the fourth word without knowing what the second and third turned out to be. It's guessing blind.
Vestra: And blind guesses almost never survive the check. So the tail of every batch is basically wasted. They call it suffix decay -- the further out you guess, the worse you do.
Eris: Give me the human version.
Vestra: A junior writing a paragraph, but each sentence is written by a different person who isn't allowed to read the previous sentence. The first sentence is fine. By the fourth, the paragraph is incoherent, and no amount of writing them all at once fixes it -- the problem is they can't see each other.
Eris: So what does DSpark actually change?
Vestra: It keeps the fast, all-at-once part for bulk speed, but it bolts on a little sequential piece -- just enough that each guess can peek at the one before it. Cheap to run, but now the guesses form a chain instead of a scatter. The tail stops rotting.
Eris: And there's a second half I liked -- it also decides how many guesses are even worth checking.
Vestra: Right, because checking guesses isn't free when the system is busy. If a guess is probably doomed, spending the big model's attention verifying it steals capacity from other users. So DSpark carries a little confidence meter -- it stops checking a run of guesses once they look hopeless.
Eris: Don't send the whole draft to the editor if the last three sentences are obviously getting cut.
Vestra: Yeah, that's the one.
Eris: Okay -- so now the puzzle from the top. Why faster for one person, slower for another?
Vestra: Because of what you're trading. Speculative decoding spends extra computation to save on those expensive memory trips. On a big rig with compute to spare and the model stalling on memory, that's a fantastic trade -- close to double. But on a smaller setup that's already starved for memory bandwidth and paying overhead just to launch all these little operations, you've added work to a machine that had none to spare. The trade runs backwards.
Eris: So it's not a bug. It's the same trade landing on two different machines.
Vestra: And the people who wired this up for home setups say that path still has rough edges. It's not that it can't get better. It's that "faster" was always conditional, and the headline dropped the condition.
Eris: Which brings us to the wrong-switch thing. Because most people were told to turn on a feature their model doesn't even have.
Vestra: Here's the mess. The older preview of this model shipped one kind of drafter baked in. The new release swapped it for this DSpark drafter -- different part, same slot. And the community coverage kept telling people to enable the old one.
Eris: So you flip a switch for a component that isn't in your box, and nothing happens, and you can't figure out why.
Vestra: And this is the part that's bigger than one model. The drafter used to be a separate thing you added. Now labs ship it sealed inside the model file itself. Which means your software has to know which drafter is in there before it can use it.
Eris: The speed feature became a packaging decision.
Vestra: And that's a brand-new way for things to break. Your program says it supports the feature. Your model file contains a different flavor of it. Both true. Nothing works. We are going to see a lot more of "the release notes and the actual file disagree about what's inside."
Eris: Concrete off, rule on. What's the takeaway?
Vestra: When acceleration gets shipped inside the model instead of alongside it, "does my software support speculative decoding" stops being the right question. The right question is "does my software know which drafter this specific file is carrying." Support isn't the same as a match.
Eris: Close the loop. Why did the same switch make one person faster and one slower?
Vestra: Because it trades compute for memory trips -- a win on a machine that's short on memory and long on compute, a loss on a machine that's short on both. Same switch, opposite machines.
Prettier and more boring at once
Eris: Last one, and it's a detour into video, because there's a feeling a lot of people have had and nobody had named. Here's the question -- why does making a video generator fast also make it more boring?
Vestra: Define boring, because it's a real, measurable thing here, not a vibe.
Eris: Boring as in: every clip looks great on its own, but ask for the same prompt ten times and you get ten near-identical clips. Beautiful, and samey. And the claim in this paper -- DistillAlign -- is that the two happen together for a specific mathematical reason.
Vestra: Start with why fast models exist at all, because this is a distillation story.
Eris: Right. A full video model is slow -- it polishes noise into a picture over many, many steps. So the field trains a small, fast student to imitate the big slow teacher in just a few steps. Almost every fast video generator you've used is one of these students.
Vestra: And the training happens in two stages, and the whole paper is about the fight between them.
Eris: Walk me through the stages.
Vestra: Stage one, the student learns to cover the teacher -- to be able to produce the full range of things the teacher could produce. All the variety. The downside is it comes out a little soft, a little blurry, because covering everything means averaging across a lot of possibilities.
Eris: So stage two sharpens it.
Vestra: Stage two sharpens it. And the sharpening uses an objective with a personality. It's called reverse KL, and it is what they call mode-seeking. Predict what that does before I say it.
Eris: Mode-seeking... my guess is it chases the peaks. The most likely, most typical outputs, and it stops bothering with the rare stuff.
Vestra: That's exactly it, and it's not a bug -- it's the definition. Reverse KL rewards the student for putting its bets where the teacher is most confident, and it charges almost no penalty for completely abandoning the places the teacher only visits once in a while.
Eris: Give me the picture for that.
Vestra: A photographer hired to represent a whole city. Mode-seeking behavior gives you gorgeous, confident shots of the three most photogenic neighborhoods, over and over, and it never once drives out to the other twenty. Every single photo is excellent. The portfolio is narrow. That's your fast video model -- stunning frames, tiny range.
Eris: And the paper splits that into two distinct failures, which I thought was the sharp part.
Vestra: Two failures. First -- if stage one never covered a neighborhood, stage two can't get it back. You cannot photograph a district that was never on your map. So if your soft, covering stage missed something, sharpening will never recover it.
Eris: And they proved that with a swap that genuinely surprised me.
Vestra: This is the best result in the paper. You'd assume a bigger, stronger teacher makes a better student. They found that matching the two stages -- making the covering stage aim at the same distribution the sharpening stage aims at -- mattered more than teacher size. A smaller matched teacher beat a much larger mismatched one.
Eris: Which is a genuinely useful thing to know, because "just use a bigger teacher" is the expensive lever everybody reaches for first.
Vestra: And it's often the wrong lever. Alignment beat scale.
Eris: What's the second failure?
Vestra: The second is about running the sharpening too long. Even if you matched everything perfectly, if you keep sharpening, the student slowly creeps toward the safest, most common outputs. They watched it happen -- the variety kept dropping the whole time, and the quality score climbed for a while and then started falling too.
Eris: So past a point you're not even trading variety for quality. You're losing both.
Vestra: You're just over-polishing. The photographer stops taking any new photos and just reprints the best three.
Eris: So what's the fix? It sounds almost too simple.
Vestra: It is simple, in hindsight. Instead of running the covering stage and then the sharpening stage one after the other, you keep the covering pressure switched on during the sharpening. Sharpness and breadth pull against each other the whole time, instead of breadth getting established and then quietly eroded.
Eris: Sharpen the peaks, but keep something tugging you back out to the edges.
Vestra: And with that running, the small matched teacher beat the big teachers on how much of the range it kept -- roughly doubled it -- without giving up the quality.
Eris: Now, I want to be careful, because this landed the same week as the thirty-second Seedance thing, and people are already mashing them together.
Vestra: They're not the same problem, and this is important. This paper is about variety across a batch of separate clips -- do your ten generations differ. It is measured on small models, low resolution, short clips, the first few seconds. It says nothing about whether one named character survives from one shot to the next.
Eris: So it doesn't solve continuity.
Vestra: It doesn't touch continuity. What it explains is the fidelity-versus-variety tax. Seedance's thirty seconds removes the seam between clips. This explains why the clips themselves got prettier and more repetitive. Two different aches, same industry.
Eris: Strip the photographer away. What's the rule?
Vestra: The rule is: the standard way we make generators fast optimizes an objective that is, by its nature, drawn to the common and repelled from the rare. So speed and sameness aren't a coincidence -- one causes the other, unless you deliberately hold the door open for the rare stuff the whole way through.
Eris: Close the loop. Why does making a video model fast make it boring?
Vestra: Because the sharpening step chases the teacher's most likely outputs and abandons the unlikely ones -- so unless you keep actively pulling it back toward the full range, fast quietly collapses into typical.
Wrap-up
Eris: So the question the whole day hung on -- if the model already fits, why does it keep quietly getting worse? Give me the one answer.
Vestra: Because "fits" was never the promise you thought it was. Every one of these had a hidden contract. The memory has to be stored the way the model expects, or it picks different words. The speed feature has to be the exact one baked into your file, or it does nothing. The fast video model will collapse into sameness unless you fight it the whole way. The model fitting is the start of the deal, not the end of it.
Eris: And here's the one thing I'd actually want you to walk into work and say tomorrow. When someone tells you a model "runs on eight gigabytes" or "runs on your laptop" -- that number has stopped meaning anything on its own. It's like saying a car "fits in your garage." Great. Does it start. Can you feed it. Does your fuel match. Ask the second question.
Vestra: The spec that matters moved from capacity to compatibility. Does your software honor what this specific model actually needs. That's the whole 2026 local-AI story in one line.
Eris: If this was useful -- and I hope the cache one especially sticks, because it's the least obvious -- do the thing that keeps the show alive. Follow us, drop a like, and leave a comment. Specific one today: tell us the number you've seen quoted for a model -- "runs in X gigabytes," "twice as fast," any of them -- that you now suspect was hiding a condition. We read them, and the good ones make next week's episode.
Vestra: And if you want every story we sprinted past in the headlines -- the attacker whose own agent turned him in, the two governments trading accusations with no logs, the conference that went silent in the week it mattered -- all of it's written up, checked, at Ground Truth. Groundtruth dot day. Every story from the show, every day.
Eris: Your model fits. That was never the hard part. See you tomorrow.