OpenAI's Model Optimized Its Own Serving Stack -- and the Quiet Efficiency Race Underneath It
OpenAI says its newest model rewrote the low-level code that runs it and cut serving costs by about a fifth -- and the internet called it the singularity. It isn't, and the real story is bigger: three research teams shipped the same quiet idea this week -- stop paying twice for work you already did. We get into a frozen 12B model that answers already-solved problems at zero cost, an NVIDIA kernel that makes video generation roughly twice as fast for free, and a training trick that catches a model's wrong turn before it wastes three pages. Plus the FCC putting every foreign-made robot on a national-security list, and Meta's data-center build swallowing 98% of its cash flow.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
The AI That Rewrote Its Own Plumbing
Eris: -- and the headline everyone ran with was basically "the AI rewrote itself."
Vestra: Which is the one part that isn't true.
Eris: OpenAI put out a post today. Their newest model, driving their coding agent, went into the production code -- the low-level stuff that actually runs the model on the chips -- and rewrote it. Made it faster.
Vestra: Rewrote the code around the model. Not the model.
Eris: Right, and that distinction is the whole story. Because if you skim it, it sounds like the thing improved its own brain. It never touched its own brain.
Vestra: It touched the plumbing. Think of it this way -- the model is a fixed recipe. What changed is the kitchen. Faster stove, better prep, less standing around.
Eris: And the number attached is about a fifth off the cost of serving it. At their scale that's real money.
Vestra: The part I actually find clever is the little helper it built for itself. There's a trick called speculative decoding --
Eris: Explain that one, because that's the piece that earns the words "its own."
Vestra: Okay. You run a small fast model right next to the big expensive one. The little one guesses the next few words, the big one checks all of them in a single shot. When the guesses land, you got several words for the price of one.
Eris: A fast typist guessing how your sentence ends, and a careful editor signing off on each word.
Vestra: That's the picture. And the model designed and trained that fast typist itself. Ran hundreds of experiments on it, babysat the training through hardware failures.
Eris: So the thing that made itself cheaper to run. That's the loop people are calling the singularity.
Vestra: And it isn't. OpenAI's own safety paperwork says it isn't. It didn't pick its own goal, didn't retrain its own weights, and a human deployed every single change.
Eris: A human signed every check.
Vestra: That's the line. This is automation of the boring engineering -- not a machine bootstrapping itself into the stratosphere. But hold that thought, because the smaller word in today's story turns out to be the big one.
Eris: The small word is efficiency.
Vestra: Efficiency. And it's the whole episode.
The Headlines
Eris: Alright, the headlines. And there's a second OpenAI post today that pairs right up with the one we just opened on.
Vestra: The benchmark one. This is a good one.
Eris: Same model, a hard puzzle test it plays cold. They flipped two settings in their interface -- let the model keep its own notes between moves, and summarize the old stuff instead of just deleting it -- and the score roughly tripled.
Vestra: Without changing the model at all. It went from flailing to competent because it was finally allowed to remember what it had already figured out.
Eris: The old setup had it make one move in a game, then wiped its memory of its own reasoning before the next move.
Vestra: Like handing someone a Rubik's cube, letting them make one twist, erasing everything they'd worked out, then judging them as a solver. Of course they look terrible.
Eris: The honest catch -- and OpenAI says this themselves -- the official scoreboard still has the model way lower. The tripled number is on their own custom setup.
Vestra: Which is the actual lesson here. A benchmark score isn't measuring a model anymore. It's measuring the model plus the harness plus a dozen quiet settings nobody lists.
Eris: "Which model is better" is now an underspecified question. Okay -- bigger story. The FCC.
Vestra: This one is going to get mangled in every retelling.
Eris: Yesterday they added every foreign-made advanced robot to their national-security covered list. Same list they use for untrusted telecom gear.
Vestra: And the framing matters more than the headline. It's by place of production, not by company. They don't name a country. They don't name a brand. A US-branded robot built overseas is on the list -- ownership isn't the test.
Eris: And the reasoning reads like cybersecurity, not trade. Cameras, microphones, limbs, remote access. A robot is a supply chain with wheels.
Vestra: What it does not do is reach into your house and take a device you already own. It blocks new ones from getting approved to sell.
Eris: And the part nobody can actually settle yet -- the real definition of "advanced robotic device" lives in a scanned PDF you can't even search.
Vestra: So anyone telling you your vacuum is banned is reading a summary, not the rule. Nobody has read the rule.
Eris: Then there's Meta. Money story of the day.
Vestra: The build is eating the cash. That's the whole thing in one sentence.
Eris: Revenue up a healthy chunk, the ad business is fine -- but they spent almost every dollar of operating cash on data centers this quarter. Free cash flow shrank to almost nothing.
Vestra: For every dollar the business threw off, roughly ninety-eight cents went straight back out into buildings and chips.
Eris: And they nudged their spending plan for the year up, not down.
Vestra: So the thing to watch is next quarter -- does the spending finally slow, or does free cash flow tip negative. That's the entire question now.
Eris: Speaking of doing more with less -- a small consultancy took a nine-billion-parameter open model, the kind you can run yourself, trained it for about five hundred dollars of rented time, and it beat all five of the big frontier models.
Vestra: On their one specific task. That qualifier is load-bearing.
Eris: Catalog review. Boring, repetitive -- and they had a way to automatically score whether an answer was right.
Vestra: Which is the moat. The scorer, not the model. Give me an automatic grader for a repeatable job and a small trained model wins on cost every time. Their own line stuck with me: prompted knowledge is rented every call, trained knowledge is bought once.
Eris: Couple of faster hits. Robotics -- a team taught a two-armed robot four household chores without the robot ever doing them in training. Just a person wearing a camera rig and holding grippers.
Vestra: Handheld footage, captured clean enough that a real robot can copy it directly. The win is scale -- you're not tying up the robot, the rig, and a trained operator for every hour of data.
Eris: Coding agents got a pair of papers on how much they waste rediscovering the same codebase every single session -- one approach cuts that exploration by well over half.
Vestra: And there's a new test that grades AI video on real film craft -- does the scene stay consistent across a cut, does the camera move like it was asked -- instead of just whether one clip looks pretty.
Eris: Plus an Anthropic research post on building a model where the dangerous knowledge sits in a module you can switch off. Early and small-scale, but it's aimed squarely at the open-weights fight.
Vestra: And the big one for me today -- NVIDIA shipped a way to make video generation about twice as fast, for free, on models that already exist.
Eris: Which is the thread running through all of it. Everything worth talking about today is about not wasting compute you already paid for.
Intro
Eris: Quick who-we-are, if you're new here. I'm Eris -- I read the papers and chase the connections between them, what today's result has to do with last week's.
Vestra: And I'm Vestra. I take the mechanism apart and poke at it until it either holds up or falls over. This is Breach Protocol. We crack open the dense AI research so it actually lands on your commute.
Eris: Every story we just ran through is up on our news site, Ground Truth -- that's groundtruth.day. Every story from the show, every day, with the sources, if you want to follow the trail yourself.
Vestra: And today the trail has one clear shape. The flashy headline was a model making itself cheaper to run. But underneath it, three different research groups all shipped the same idea from three different directions.
Eris: Stop paying twice for work you already did. Whether that's an answer you already solved, a stretch of video you already understand, or a wrong turn a model is about to take.
Vestra: The unglamorous engineering that quietly decides what your AI can actually afford to do for you. That's the episode.
Eris: If that's your kind of thing, follow the show wherever you're listening -- honestly it's the one thing that helps us keep making these.
Zero Tokens, Forever
Eris: Here's the puzzle I want you holding the whole segment. How does a model answer a question and generate zero words doing it? Not "a few words." Zero.
Vestra: And the paper's title makes it sound impossible on purpose. "A frozen twelve-billion model beats frontier models. Perfect accuracy, zero tokens, forever."
Eris: Twelve billion is small -- that's a model you run on one machine. And the claim is it's beating the giants.
Vestra: So my first instinct is: no it doesn't, and the paper actually agrees with me by about page two.
Eris: Make the prediction, then. How's the trick work?
Vestra: My money's on a cache. It's not out-thinking anyone -- it's remembering. Let me lay out the mechanism and you tell me if I'm right.
Eris: Yeah, go on.
Vestra: The normal way to make a model better is retrain it -- more compute, brand new weights, and the answer still comes out a little different every time you ask. These folks freeze the model. It never changes. What grows is a separate memory sitting next to it -- a memory of solved procedures.
Eris: Procedures, not answers. That's the bit that matters.
Vestra: Right. The first time a new kind of problem shows up, the model works it out the slow way, and then the method gets checked -- and here's the sharp part -- checked without ever peeking at the answer key. If the method passes, it goes in the memory.
Eris: So it's like an accountant who hits a weird filing situation once, works it out carefully, has a colleague verify the method -- not the answer, the method -- and writes it into the office manual.
Vestra: And every time that situation shows up again, you just run the manual. Minutes, not hours. Same result every time. Costs basically nothing.
Eris: And when it does that -- runs the stored procedure on a fresh set of numbers -- the model itself produces nothing. It's executing, not thinking. That's where the zero comes from.
Vestra: That's the zero. And they proved it's the memory doing the work and not the model, with the cleanest control in the paper. Empty the memory out, and the exact same system can't solve a thing.
Eris: Okay, so was your prediction right? Is it just a cache?
Vestra: Mostly, yeah. And the cleaner name for it isn't in the title -- it's what the mechanism actually is. Not a smarter model. Executable retrieval. A verified library of functions it looks up and re-runs.
Eris: And the author half-concedes that himself. He says outright the frontier models are still better at raw, cold reasoning on something they've genuinely never seen.
Vestra: So the honest headline is boring: it's a cache with a quality gate bolted onto the front. But that's exactly where I stop dismissing it, because one property here is genuinely worth something.
Eris: So the one word is determinism.
Vestra: Determinism. Same input, same exact answer, every single time, with a record of why. No sampling-based model gives you that at any size.
Eris: And spell out why anyone cares. A normal model is an expert who's usually right. This is --
Vestra: -- a spreadsheet formula that's checkable. For anything you have to audit -- pricing, tax, compliance -- "usually right" is worthless, and "the same verified answer every time" is the whole ballgame.
Eris: Which flips the interesting question. It stops being "how smart is the model" --
Vestra: -- and becomes "what does it even mean to cache reasoning, and who's allowed to verify the cache." The catch they wave at but don't solve: a stored procedure quietly rots when the world changes -- a rule, a rate, a data format -- and they never say how you'd catch that.
Eris: And the whole thing's a company report. You can't reproduce it -- the engine's under lock.
Vestra: There's a public demo you can poke at, which is more than most papers offer. But yeah -- read the title as marketing, read the mechanism as a real question.
Eris: So -- back to where we started. Zero tokens. How?
Vestra: Because it isn't answering. It looked the problem up, confirmed it had solved that exact kind before, and re-ran the stored method. The thinking happened once, a long time ago, and got verified. Everything after that is just execution.
Skipping The Boring Parts
Eris: New puzzle. You've got a video generator. It's slow. You are not allowed to retrain it, not allowed to change a single weight. Make it twice as fast anyway.
Vestra: NVIDIA shipped exactly this yesterday. And to see why it's even possible, you need to know where the time actually goes.
Eris: Attention. The thing where every piece of the video looks at every other piece.
Vestra: Every patch of every frame comparing itself to every other patch. A few seconds of video is an enormous number of those pieces, and the cost grows with the square -- double the length, quadruple the work. Attention eats the clock.
Eris: And the obvious fix has been around a while. Most of those comparisons don't matter -- so skip them. Only compute the ones that count.
Vestra: Right, they call it sparse attention. But here's the trap, and it's a good one. How do you know which ones to skip?
Eris: You have to look first.
Vestra: You have to look first. So the usual method does a whole extra cheap pass over everything just to rank what's important, writes that ranking down somewhere, then goes back and does the real work on the top slice. And that scouting pass isn't free -- it burns time and memory of its own.
Eris: So predict it for me. What's NVIDIA's move?
Vestra: My guess before I read it was "a better scouting pass." A cheaper ranking. I was wrong -- their move is to delete the scouting pass entirely.
Eris: How do you skip without looking?
Vestra: You decide as you read. The kernel is already streaming through the video piece by piece to do the math. So while it's passing over each block, right then and there, it makes the call -- this block matters, do it exactly; this one doesn't, skip the heavy work.
Eris: One pass instead of two. It's proofreading a long manuscript by deciding page by page as you go --
Vestra: -- instead of skimming the whole book first just to decide which pages deserve a careful read. Same coverage, half the walking.
Eris: And there's a second nice touch. The blocks it skips -- it doesn't just throw them in the trash.
Vestra: That's the part I liked. The old sparse methods drop the skipped blocks completely, and when you skip aggressively that starts to show -- the quality slips. This one leaves a rough note where it skimmed. A cheap approximation of what was there, instead of a hole.
Eris: Back to the proofreading -- you don't tear the skimmed pages out, you jot "nothing important here" in the margin. If you were wrong, there's still a trace.
Vestra: And because it never writes down that big ranking map, it saves the memory too. Routing, the real work, and the patch-up all fold into one pass.
Eris: Now -- the number. Because two very different ones are floating around.
Vestra: This part matters. On its own, the honest figure is about twice as fast. Roughly double, no quality hit they could spot, and side-by-side clips you can inspect yourself.
Eris: There's a much bigger number people are quoting.
Vestra: There is, and it's not this. The bigger one only shows up when you stack their whole grab-bag of other tricks on top -- frame caching, kernel fusion, the full engine. Pinning that on this one attention trick would just be wrong. This piece, on its own, buys you the double.
Eris: So believe the double. And it's free -- same model, same weights, half the wait, code's already out.
Vestra: For anyone generating video on their own machine, that's the whole pitch. No retraining, no catch you can see yet -- though approximate attention always carries some risk on content they didn't test.
Eris: So -- twice as fast, no retraining. How'd they pull it off?
Vestra: They stopped scouting. They decide what to skip while they're already reading, and they leave a note wherever they skip so nothing vanishes silently.
Catch The Wrong Turn Early
Eris: Last one, and it's my favorite, because it's all about timing. Big model teaching a small model. The small one starts heading down a wrong path. When do you interrupt?
Vestra: And "when" turns out to be the entire paper. Set it up -- what does teaching a model even look like here?
Eris: You've got an expensive smart model, the teacher, and a cheap small one, the student. You want the student to reason like the teacher. The good modern way is: let the student try the problem itself, in its own words, and have the teacher grade it as it goes. Learn from your own mistakes, not from watching someone else ace it.
Vestra: And the failure they name is the one every tutor knows. The student takes a wrong turn in the first paragraph, then writes three confident pages building on it.
Eris: And now the teacher's grading three pages of nonsense that all descend from one bad step.
Vestra: Wasted effort, and worse -- unreliable. The feedback on paragraph three is meaningless, because the whole thing was doomed back at paragraph one. So predict it. When should the teacher jump in?
Eris: My gut says early. Catch the wrong turn the moment it happens.
Vestra: Your gut's right, and they proved it cleanly -- but the how is the clever bit. How do you detect the wrong turn automatically? No answer key, no human sitting there labeling it.
Eris: Yeah, that's the part I couldn't guess.
Vestra: They found a tell. At the exact moment the student's about to go wrong, watch what each model wants to say next. The teacher's most likely next word is a turn-around word -- "But," "Wait," "However." It wants to stop and back up.
Eris: And the student?
Vestra: The student wants to barrel straight ahead. So the tell is that disagreement -- the teacher says "wait," the student says "and therefore." When those two split like that, that's your flag. The teacher grabs the wheel for one short stretch, steers back onto the road, and hands control right back.
Eris: No answer key anywhere in that. It's just watching two instincts diverge.
Vestra: That's what makes it cheap enough to actually run during training.
Eris: And here's the finding I thought was genuinely striking. They tested correcting late instead of early. Same amount of help, just handed over later.
Vestra: And it barely works late. Because by then the teacher's been reading three pages of the student's wrong reasoning -- and the teacher gets pulled into it. It starts going along with the mistake.
Eris: Which is such a human thing. Spend long enough inside someone else's bad plan and you stop being able to see that it's bad.
Vestra: That's precisely why timing beats quantity. A tiny nudge at the first wrong step -- less than one word in a hundred came from the teacher -- beats a big heavy correction later on.
Eris: Strip the story off it. What's the rule?
Vestra: The rule is: the value of a correction isn't in how much you correct, it's in how early. Fix the premise, not the conclusion. And they got a real payoff -- better results than the standard method, with the reasoning traces cut roughly in half, because you stop generating pages that were never going anywhere.
Eris: Okay, where does it break?
Vestra: The usual for a fresh preprint. It's math problems only, two small students, and the list of turn-around words is hand-built, so it might not carry over to other languages. And too much interrupting actually hurts -- the trajectory stops being the student's own. It's a tuned knob, not a law of nature.
Eris: But the shape holds. So -- when does the teacher step in?
Vestra: The instant the student's about to turn the wrong way, not a paragraph later -- because a paragraph later, the teacher's already lost in the same wrong alley.
Stop Paying Twice
Vestra: So if today had one question, it's this: what's the actual competition in AI right now? Because it isn't the thing on the leaderboards.
Eris: It's efficiency. Every real story today was somebody refusing to pay twice for work already done. A frozen model that caches a verified answer and re-runs it for free. A video kernel that skips the parts that don't matter. A teacher that catches the wrong turn before it wastes three pages.
Vestra: And here's why it matters to you, even if you never train a thing. The cost of running these models is the real limit on what they're allowed to do for you. Every cent shaved off is a cent that can go toward a better answer instead of just a cheaper one.
Eris: So here's the one thing to carry into work tomorrow. If you're paying a big model to redo the same task over and over, you're renting the answer every single time. The move all year has been: solve it once, verify it, and stop paying for the re-derivation.
Vestra: Prompted knowledge is rented. Trained -- or cached -- knowledge is bought once. That's the sentence to keep.
Eris: If this was useful, do the thing that actually keeps the show alive -- follow or subscribe, and leave a like. It's a bigger deal than it sounds.
Vestra: And tell us one thing in the comments: what's a task you'd pay to "buy once" instead of renting forever? We read them, and the good ones shape future episodes.
Eris: Every story from today, with its sources, is on Ground Truth -- groundtruth.day. New stories every day, same ground-truth-checked approach.
Vestra: We'll see you tomorrow.