The $15 Frontier Run: When the Harness Beat the Model
A team wrapped last year's model in a smarter harness, weights frozen, and edged out the newest frontier model on a live coding benchmark -- then did it again with a cheap model for about fifteen dollars, matching a frontier run that cost nearly six hundred. The trick wasn't the model. It was the cage around it. We break down three papers from one week that all say the same quiet thing: the fastest gains in AI right now aren't inside the model -- they're in the scaffolding that keeps it on track, trains it, and grades it. Before you pay for a bigger model, look at your harness.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
The $15 record
Eris: Somebody just beat the newest frontier model on a live coding benchmark -- using last year's model. Didn't train a thing.
Vestra: Define beat for me.
Eris: Edged it out. Weights frozen, not one gradient touched -- last generation, just run smarter.
Vestra: So they spent the training budget somewhere--
Eris: --Nowhere. There was no training budget at all.
Eris: And in a separate run they took a cheap model, never trained that one either, and reached the same frontier for about fifteen dollars. The system it matched cost nearly six hundred.
Vestra: Okay, that gap is not a rounding error. What did the fifteen dollars buy?
Eris: The cage around the model. Not the model. And once you see what a cage can do, "just buy the bigger one" starts to sound like the expensive answer.
Vestra: Then let's open the cage.
The $15 record, no new model
Vestra: So here's the question I actually want answered. If the model never changed, why did it suddenly stop failing?
Eris: Right, because that's the weird part. The paper's whole starting point is a failure that shouldn't happen. You give a long agent job to a model that can genuinely do every single step. And it still blows the run.
Vestra: Which sounds like a contradiction until you've watched one of these agents work for twenty minutes.
Eris: Walk me through it.
Vestra: It loses the plot. Literally. It's doing a fifty-step task, and somewhere around step thirty it forgets what it already finished, redoes something, skips a check it did perfectly an hour ago, or just... decides it's done when it isn't.
Vestra: Every one of those steps, on its own, the model can nail. The job still fails.
Eris: So the model isn't the thing that's broken.
Vestra: The model isn't the thing that's broken. What's broken is that the whole task state -- what's done, what's pending, what already failed -- is buried in this enormous scroll of everything it's ever said. And the plan you gave it at the start is way up at the top of that scroll, getting quieter and quieter as the transcript grows.
Eris: Okay, so before I hear the fix -- my guess is they just remind it more often. Re-paste the plan. Nag it.
Vestra: That's the obvious move and it mostly doesn't work, because you're adding more text to the pile that's already drowning the signal. What this team did is stranger. They stopped treating the job as one long conversation and started treating it as a set of rooms.
Eris: Rooms, like actual rooms?
Vestra: Rooms. They call them states. Think of the task as a building with a few rooms -- plan, build, check your own work, repair, hand off. The model walks through them in order. And every time it walks into a room, the room hands it a fresh index card: here's the phase you're in, here's what you need to do here, here's where you actually are.
Eris: So it's not re-reading the whole scroll to figure out where it is. The room just tells it.
Vestra: The room just tells it. That's the entering half. The leaving half is the part I actually like -- you can't walk out of a room until you pass the checks on the door.
Eris: Meaning the model can't just announce it's finished.
Vestra: It cannot declare victory and stroll out. To leave the "build" room, the door might run the actual tests. Run a real command. And if that check fails, the door stays shut -- it's stuck in that room until it fixes the thing.
Eris: And this is the piece that connects to something you say on this show constantly. A model saying "done" is not evidence that it's done.
Vestra: It is not, and this system is unusually honest about that. It sorts its checks by how much you should trust them. A check that runs a command and reads the result -- that's real, reproducible evidence. A check that's just the model ticking a box that says "yes I did it" -- that's the model's word, and they label it as exactly that. Weaker. A second model reviewing it -- better than nothing, still not proof.
Eris: So it's not pretending a checkbox is a fact.
Vestra: It refuses to launder a checkbox into a fact just because it sits in an official-looking field. That restraint is rare.
Eris: Let me pull the thread you dropped a second ago, though, because I think it's the actual engine here. You said "rooms." Where do the rooms come from? Who writes the building?
Vestra: This is the good part. The building is just a plain text file. A runbook. Human-readable, sitting right there in the workspace. And both the model and the person can read it, edit it, version it. It's not hidden controller code off to the side -- it's a shared document they both operate.
Eris: And that shared document is what makes the whole thing improvable. Here's the loop, tell me if I've got it. Agent runs the job. It fails somewhere. Afterward, somebody -- or another agent -- looks at the failure and asks: was this missing information? A door that should've been locked and wasn't? A check that was too weak?
Vestra: Right, they've got a little taxonomy of failure types.
Eris: -- and then they patch the building. Add the check. Tighten the door. And that patch is saved, versioned, and it's there the next time. So the model learns nothing --
Vestra: -- but the system does. Exactly that. The lesson lives outside the weights, in the runbook.
Eris: That's the sentence. There's this gap they name -- the model correctly figures out what went wrong, and then completely fails to remember that lesson the next time the same trap shows up. The runbook is the memory the model doesn't have.
Vestra: And notice it's a different kind of memory than "here's more stuff to read." It's procedure. Not "here are facts," but "at this exact door, do this check, every time, forever." That's the thing an agent forgets, and that's the thing they made permanent.
Eris: So who feels this? Because in the abstract it's an architecture diagram. Who's it for?
Vestra: Anyone paying for the biggest model on reflex. That's the stakes. The reflex right now is: my agent's flaky, buy a stronger model. This paper's whole argument is that a big chunk of that flakiness was never the model. It was the scaffolding. And scaffolding is cheap.
Eris: Which is where the numbers get almost rude. Take the older-generation model, wrap it in this, and it climbs a full model-generation's worth of quality -- enough to edge out the next-gen frontier system. Same model. Just a better cage.
Vestra: And the cage travels. They built it on one model, froze it -- didn't touch it -- pointed it at the newer model in the same family, and it just worked. Carried the lesson forward for free.
Eris: The cheap-model story is the one that'll get quoted, though. They took a budget model, adapted the cage a little, and got it up near the expensive frontier result -- and the final bill for the winning evidence was about the price of a sandwich. The model it matched cost a nice dinner for four. Same benchmark.
Vestra: And here's where I have to put the brakes on, because they do too, to their credit. That headline record is a raw score on an open submission that hasn't been accepted yet. Some of the winning runs got flagged for possible gaming -- the agent finding a cheap trick instead of really solving it -- and the honest lower numbers are lower.
Eris: So the record has an asterisk.
Vestra: It has an asterisk, and they printed the asterisk themselves. Second thing: the cage didn't transfer for free everywhere. Move it to a totally different provider's model and the exact settings broke -- they had to redo the specifics. What travels cleanly is the structure and the method, not always the fine print. It's a reusable artifact, not plug-and-play.
Eris: That's fair, and it actually makes the claim stronger, not weaker, because the claim was never "the model doesn't matter."
Vestra: No. The claim is the model and the scaffolding are two separate dials, and almost everyone's been cranking only one of them. In these tasks, the scaffolding dial had more room left in it.
Eris: So let me strip the building away and say the flat version. Forget rooms and runbooks and doors.
Vestra: Yeah, please do.
Eris: The model is the actor. It can play any single scene brilliantly. What it doesn't have is a stage manager -- someone tracking which scene we're in, holding it to its marks, and not letting it walk off until the scene actually lands. The whole paper is: hire the stage manager. It's cheaper than recasting.
Vestra: And you can keep the same actor.
Eris: So -- back to your opening question. The model never changed. Why did it stop failing?
Vestra: Because the failures were never about knowing how. They were about staying on track, remembering the lesson, and not quitting early -- and every one of those is something you can fix in the scaffolding for the price of a sandwich, without touching the model at all.
Teaching the model to use the cage
Eris: So if the last story is "build a better cage by hand," here's the mirror image. Different team, same week. Their question is: can you train the model to use the cage -- when you can't see inside the cage?
Vestra: Wait, back up. Why can't you see inside it?
Eris: Because the cages people actually run in production are these big commercial systems -- the coding harnesses, the assistant harnesses. And they're not simple loops. Under the hood they fork, they retry, they spawn little helper agents, they quietly manage what the model sees.
Vestra: So from the outside it's a black box. You put a task in, work happens in there, an answer comes out.
Eris: A black box. And that's a real problem if you want to train the model to be good inside that box. Because the standard way you train these things with reinforcement -- reward the good runs, discourage the bad ones -- needs a clean recording of what the model actually did. Step, step, step.
Vestra: And the box doesn't give you--
Eris: --that. Not a clean one. What you get, if you peek, is a mess. Fragments. Forked-off calls. The same thing tried three times. Redundant junk. It's not a clean trajectory, it's shrapnel.
Vestra: The tempting move here is to crack the box open. Rewrite the harness so it hands you clean training data.
Eris: That's the tempting move and it's a trap, because then you've trained your model on a harness that isn't the real one. The whole point was to be good inside the actual production system, forks and retries and all.
Vestra: So they leave the box shut.
Eris: They leave it shut. And this is the clever bit. They slip a recorder in at the one place everything has to pass through -- the boundary where the harness talks to the model.
Eris: Every single time the box needs the model to think, that request goes through their little checkpoint. So they don't need to understand the box. They just catch every word going in and every word coming out.
Vestra: Ah. So you reconstruct the run from the outside. You never see the machinery, but you see every conversation the machinery had with the brain.
Eris: Exactly. And then they take all those scattered, forked-off calls and reassemble them into a tree -- because the box tried three branches, and a tree is the honest shape of "here's the path that worked and here are the ones that didn't." Then they train over the tree.
Vestra: Let me sit on the tree for a second, because I think that's carrying more than it looks. When the harness forks -- tries three approaches -- a flat recording would just look like noise, three overlapping transcripts. The tree keeps the branching structure. So the training can actually tell "we were at this decision point, and this branch is the one that paid off."
Eris: The shape of the retrying is information, and they keep the shape.
Vestra: And here's the part that's genuinely hard, not a footnote -- keeping the training honest. There's this failure mode where what the model saw during the run and what your training pipeline thinks it saw drift apart, and the whole thing gets unstable and falls over. Long agent runs make that worse, because tiny mismatches pile up over hundreds of steps.
Eris: So a lot of the work is just... not letting it wobble.
Vestra: A lot of the real work in these papers is boring stability plumbing, and it's the difference between a method that works for ten steps and one that holds for hundreds. They report it stays steady over a long training haul. That claim is the paper, honestly, more than any single score.
Eris: And the payoff number is real. Same starting model, run it through this, and it gets meaningfully better at working inside those harnesses -- a bigger jump through the more demanding coding harness than the lighter assistant one.
Vestra: Which, side note, is a nice sanity check. The harder cage had more headroom to teach into. That tracks.
Eris: And then the move I think is the actual headline. They don't train one model for one cage. They mix them. One model, trained through several different harnesses at once.
Vestra: And that shouldn't obviously work. Different cages, different tool setups, different rhythms.
Eris: Shouldn't work, does work -- the mixed one matches or beats the versions trained on a single harness alone. So you're not fine-tuning a model per box. You're growing a model that's just... generally good at living inside boxes.
Vestra: So who's this for? Because the last story was for the person picking a model off a shelf. This one isn't.
Eris: This one's for whoever's building the box. If you're the lab shipping the harness, you now have a way to train your model to actually exploit it -- without hand-writing every rule the way the last team did, and without the model ever needing to see your secret sauce.
Vestra: Two directions on the same road. One team hand-crafts the cage. This team lets the model learn the cage from the outside.
Eris: Give me the flat version, no tree, no boundary.
Vestra: Here it is. You can coach an athlete without the opponent's playbook. You film every play -- what they ran, what came back -- and you train off the film. You never need to see the other team's huddle.
Vestra: The signal was never in the machinery. It was in the record of what the model said and what the world said back.
Eris: So -- can you train a model to use a cage you can't see into?
Vestra: Yes, if you stop trying to see in. Stand at the doorway, record everything that crosses it, rebuild the run from that -- and the box stays a box the whole time.
Even the judge got a cage
Eris: So we've built the cage, and we've trained the model to live in the cage. There's one player left who's still just... handing out a number. The judge.
Vestra: Meaning the benchmark.
Eris: Meaning the benchmark. And here's the question this last team opens with, which I think is the sharpest one on the board today: what is a score actually worth if it can't tell you why?
Vestra: What are they judging here?
Eris: Their target is world models -- the systems that try to generate a little interactive world, a video you can act inside, push an object, walk around, come back. And judging one of those is brutal. You have to know: did the physics hold? Did the thing you shoved actually move right? Is the room still the same room when you turn back around?
Vestra: And a human spots all of that instantly. You watch the clip and you just see the cup fall up instead of down.
Eris: Instantly. But the machines grading these things can't explain a thing. They grind out a score by brute force, and there's no reasoning you can look at. No chain. Just: here's a six. Trust me.
Vestra: Which, if you think about it, is exactly the checkbox problem from the first story, one floor up. A number in an official field, no evidence behind it.
Eris: That's the connection, yeah. And their fix is the same shape as everything else today. Don't hand me a rubric with one number at the bottom. Turn the judge into an investigator.
Vestra: What does "investigator" actually mean here, mechanically? Because I can smell a version of this that's just "add a chatbot judge," and that's not new.
Eris: It's more than that. There's a lead agent that looks at the specific clip and asks, what does this case even need me to check? Then it breaks that into small, concrete, measurable questions and hands each one to a specialist.
Vestra: Give me the collision example, because that's the one that made it click for me.
Eris: Perfect one. Two things are supposed to bump into each other. The lead doesn't eyeball it.
Eris: It spins up little helpers: one traces where each object is, frame by frame. One checks -- did they actually occupy the same spot at the same moment, or just look close? One estimates how fast they were going.
Vestra: So "did the collision look real" becomes three boring, checkable measurements.
Eris: Three boring measurements, and then the lead agent gathers up what they found, sanity-checks it, and only then gives a verdict. And the thing you get out isn't a number. It's a tree of evidence. You can walk down it and land on the exact sub-question that failed.
Vestra: And that's the whole pitch, right there. Not a better number -- a number you can audit. When the model fails, you can point at where.
Eris: And it's disciplined about what it doesn't check. If a case doesn't call for a certain test, it skips it -- and writes down why it skipped it. So you're not padding the score with measurements that didn't apply.
Vestra: That's the part that separates it from a model just vibing out an opinion. It records the reason for every probe it ran and every one it passed on. The skips are on the record too.
Eris: They pointed it at a big spread of these world models, a few hundred cases, and its ranking lined up well with what people actually prefer -- which is the bar for a judge. But the diagnosis is the point. It found something a scalar would've completely hidden: when you take a plain video generator and retrofit it into an interactive world, it doesn't just get uniformly better.
Vestra: It moves the lump under the--
Eris: --rug. It gets better at some things and quietly worse at others. And a single score would've blended that into one bland average and told you nothing. The tree shows you the tradeoff.
Vestra: So who needs this? Because "better benchmark for world models" sounds narrow.
Eris: It sounds narrow and I'd argue it's the least narrow thing today. Anyone who's ever pointed at a leaderboard and said "this model is better" is trusting a number they can't inspect. Every one of those numbers has the checkbox problem. This is a template for making the grade show its work.
Vestra: And it closes a loop across all three of these, doesn't it. The scaffolding idea started as a way to make a model finish a job. Then a way to train the model. And now it's crawled all the way into how we judge them.
Eris: The judge got a cage too. Strip it to the flat version for me.
Vestra: Here's the flat version. A score you can't question is a rumor with a decimal point. Whether it's an agent saying "task done," a leaderboard saying "this model's better," or a grader saying "the physics held" -- if there's no chain of checkable evidence under it, you don't actually know anything. All three papers today do the same one thing: they replace somebody's word with something you can walk back and verify.
Eris: So -- what's a score worth if it can't tell you why?
Vestra: About as much as the checkbox. The value was never the number. It's the evidence trail you can follow when the number's wrong.
The dial nobody was turning
Eris: So the question the whole episode's been circling: does a better score mean a better model?
Vestra: And the answer today is: less and less. Three separate teams, one week, all saying the same quiet thing.
Vestra: The biggest gains right now aren't inside the model. They're in the software wrapped around it -- the part that keeps it on track, the part that trains it, the part that grades it.
Eris: And if you want one thing you can actually use tomorrow, here it is. Before you pay for a bigger model because your agent is flaky -- look at the scaffolding first. A lot of what looks like the model being dumb is the model being unmanaged. The stage manager is cheaper than the movie star, and this week somebody proved it for about the price of a sandwich.
Vestra: Which is also the uncomfortable footnote for anyone betting the farm on scale. If the fastest wins are in the harness, then pausing the training run costs you less than it looks like.
Eris: That's the thread we didn't fully pull today, and it's worth--
Vestra: --sitting with. But that's a bigger fight than we're finishing tonight.
Eris: Every story we touched is on our news site, Ground Truth -- that's groundtruth.day -- where we track this stuff every single day, and there's a lot more that moved this week that we didn't get to here.
Vestra: The rest of today's headlines are over in our AI News Today brief -- a protein-design result that two labs actually built in a lab, a leaderboard fight over robots, all of it. Go grab that one next.
Eris: One favor before you go -- drop a comment with the flakiest AI agent you've dealt with, and whether you think it was the model or the cage. We read them, and we want to know which way you land.
Vestra: Follow the show so the next one finds you, leave a like if the stage-manager thing reframed anything for you, and we'll see you tomorrow.
Eris: Look at your harness before you look at your wallet.