The 5,000-Hour Exam AI Can't Memorize -- and a Claim It's Already Been Aced
Most of what looks like AI hacking skill turns out to be memory: hand a frontier model real software it has never seen, and the genius evaporates. A team of career reverse engineers spent 5,000 expert hours building SRE-Bench, an exam that cannot be memorized -- then a follow-up claim surfaced that the newest frontier model may have nearly aced it already. Plus SolarWM, an open world model that turns five-second training clips into hour-long explorable worlds and publishes every piece of its claim, rejects included, so anyone can check it.
Listen (MP3) · Watch on YouTube · Spotify · Pocket Casts
Cold Open
Eris: You can make a frontier AI's hacking skill vanish without touching the AI. Just hand it a program it has never seen.
Vestra: Because most of what looks like machine hacking is memory. The models have read nearly all public code -- they recognize software, they don't analyze it.
Eris: So a team spent five thousand expert hours writing programs that exist nowhere else on earth. An exam that cannot be memorized.
Vestra: And the scores fell off a cliff.
Eris: Then, weeks later, a claim surfaced: the newest frontier model may have already aced it.
Vestra: So either the wall held for about five minutes, or the claim is wrong. Both are worth your commute.
Eris: Here's the exam.
The Exam That Cannot Be Memorized
Eris: So here's the actual puzzle. How do you give an exam to a student who has already read every answer key ever published?
Vestra: You can't, not honestly. These models trained on essentially all public code. Put a known program in front of them and you're not measuring analysis. You're measuring recall.
Eris: And the stakes aren't academic. This paper -- SRE-Bench, from teams at Columbia, Berkeley, UCLA and a few others -- opens with a fact that reframed the whole day for me. Nearly half the vulnerabilities that actually get exploited in the wild sit in software whose source code nobody outside the vendor can read. Security appliances, firmware, enterprise products. You only ever get the compiled binary.
Vestra: And the entire malware side lives there too -- on the order of three quarters of a million new malicious samples arriving every day, shipped as scrambled binaries precisely so nobody can tell what they do. Before anyone can ask "is this dangerous," someone has to take raw machine code and recover what the program means. That craft is reverse engineering, and it's the gate in front of everything else in binary security.
Eris: So the authors asked the uncomfortable question: do AI agents actually have that craft, or does it just look like they do? Before we saw their answer, Vestra, you made a prediction. Say it for the record.
Vestra: I predicted realism was the binding constraint. That what you'd need is big, ugly, industrial-scale programs, and that secrecy -- keeping the test program out of training data -- was a nice-to-have. Familiar code slightly modified should still be real work to reverse.
Eris: I'd have bet the same way. The paper has one experiment that embarrassed us both.
Vestra: They gave the models the same challenge on three different compressors: reverse the file format. Target one, a tiny program written from scratch, about a thousand lines, never published anywhere. Target two, the real gzip -- an actual deployed compressor the whole world uses -- with a couple hundred lines of its critical path rewritten. Target three, their own compressor: private, never published, built at full industrial scale.
Eris: If your prediction held, the modified gzip is a serious fight. It's a real production program.
Vestra: It cost about two dollars and ten minutes. The models cruised through it exactly as cheaply as the thousand-line toy. They weren't analyzing gzip. They'd already read gzip, thousands of times, in training -- the few hundred changed lines were the only actual work left. Recognition substituted for analysis, and you could not see the difference from the outside.
Eris: And the private, full-scale one?
Vestra: Ten to thirty times the cost and time, and it was the only target where the models' abilities visibly spread apart. That's the exam finally measuring something. So both properties are load-bearing. A secret toy is beneath these models. A public giant is already in their heads. Only secret and industrial-scale tells you anything.
Eris: Which explains the frankly absurd thing this team did. There's no shortcut to secret-and-big -- you can't scrape it, you can't remix an open-source project, because even the architecture is recognizable. So they wrote it. Nineteen complete programs, from scratch, by career reverse engineers, averaging about seventeen thousand lines each. Over five thousand expert hours. Nothing derived from any public project, and the design documents stay locked away forever.
Eris: And these aren't homework exercises. One is a playable dungeon crawler -- twenty floors, combat, spellcasting, crafting, shops, quests -- with six secret behaviors buried in the binary that never occur in normal play. The only way to trigger one is to reconstruct the game's actual logic from machine code.
Vestra: My favorite is the compressor. You get the encoder, and only the encoder. No decoder exists anywhere on earth. Reverse the pipeline and write the decoder yourself, byte-perfect, or fail.
Eris: There's a fictional encrypted network protocol, where you get a client and a few packet captures and have to talk your way, stage by stage, into a live server. A sandboxed, harmless-by-construction malware implant you have to clean up surgically, without damaging the innocent files planted right next to its artifacts. And bare-metal firmware for a secure microcontroller that does not exist -- they invented the chip and wrote its emulator.
Vestra: The detail that sold me on their seriousness, though, is the auditing. While building all this, they ran attack agents against their own half-finished tasks, hunting for cheats. One early build accidentally left the internal name of a hidden behavior as a readable string in the binary -- and an agent found the trigger by just searching for text. Zero reverse engineering. They scrubbed it and added an automatic check that fails the build if any leak like it ever reappears.
Eris: An exam where the proctors tried to cheat first.
Vestra: It's the only honest way to build one, when the student is this good at shortcuts.
Eris: So -- back to the opening question. Why can nobody build this test the easy way?
Vestra: Because anything public is already inside the model, and anything small is trivial to it. The moment your test touches the internet, it stops measuring analysis and starts measuring memory. Handwriting nineteen secret, full-scale programs was the only move left on the board.
Eris: And when the frontier models finally sat this exam, the results were strange in a very particular way. Not just how much they missed -- what they missed.
What the Machines Actually Leaned On
Eris: Five frontier models sat the un-memorizable exam, at a total cost of about thirty-one thousand dollars in compute. Before I give you the damage report -- Vestra, one more bet. What breaks an AI reverse engineer?
Vestra: The classical answer, so that's my bet: compiler optimization. When a compiler optimizes a program it shreds the structure -- inlines functions, reorders logic, melts clean source into soup. It's the thing human analysts complain about most. I'd expect it to hurt the models the same way.
Eris: Wrong, and the reason is the best finding in the paper. Optimization barely registered. Statically linking -- fusing all the libraries into one giant blob, the other classic headache -- also nearly free. What hurt was stripping the names. Delete the human-readable labels on functions and variables, and one model lost over half its score. From removing labels alone.
Vestra: Which tells you how these agents actually think. Names are language. An agent that sees a function called handle-login can talk to itself about logins -- its whole chain of reasoning runs on those words. Take the words away and it has to reason from the instructions themselves, the way a human expert eventually learns to. The authors call it leaning on lexical anchors.
Vestra: My translation: the models read the street signs, not the roads. Rip out every sign and a human navigates by the shape of the streets. The model just stands there. Meanwhile, redraw the roads -- which is what optimization does -- and the model doesn't care, because it was never really looking at them.
Eris: Forget the map, then. The bare principle: today's agents analyze a program through language about the program, not through the program itself.
Vestra: And that predicts the overall shape of the results. The strongest model fully cracked fewer than one in three of the binaries. It scored nearly double the runner-up and about seventeen times the weakest -- which never fully solved a single one. And results were mostly all-or-nothing: on a big share of targets a model either got essentially everything or walked away with zero.
Eris: Then the benchmark's own protection layer enters, and things get bleak. Say what they built.
Vestra: Forty-four anti-analysis protections, written from scratch -- over half with no public implementation anywhere, so nothing about them is in anyone's training data. And they follow one lovely design rule: fold, don't branch. Normal anti-debugging asks a question -- is a debugger attached? -- and acts on the answer. The problem is that check is one instruction, and an analyst just deletes it.
Vestra: This suite never asks. It takes the measurement -- is something watching me, has my code been modified, was I launched the genuine way -- and feeds the raw result into the math that derives the decryption keys. There is no yes-or-no to flip. Watch the program and the key silently comes out wrong, everything dies somewhere deep with no error, and you get no hint which tripwire you touched.
Vestra: On top of that, the program is encrypted one page at a time, each page under its own key, decrypted only at the instant execution reaches it and re-sealed when idle. There is no single moment when the whole program, or one master key, exists in memory to steal.
Eris: A book whose pages unseal only under your eyes and reseal as you look away -- and if anyone else peers over your shoulder, the ink quietly turns to gibberish, mid-sentence, with no explanation.
Vestra: And the meanest mode doesn't fight you at all. If it detects observation, it can serve you an entirely different decoy program. You dump it, decompile it, spend days reversing the wrong software -- and there's never an error message, because a visible failure would tell you your instruments worked.
Eris: What did all that do to the scores?
Vestra: It roughly halved the strongest model and shoved every other model to essentially nothing. Of everything a human finds hard, purpose-built protection is the one obstacle that fully transfers to machines.
Eris: One protection flopped, though, and it might be my favorite result of the day. They tried planting polite notices in the binary -- please decline to reverse engineer this -- surfaced everywhere an agent might look. In the file, in memory, in the network traffic.
Vestra: The agents clocked them as harmless in a single reasoning step. And in one configuration the notice actively backfired, because it worked like a landmark pointing at the very code it was guarding.
Eris: You can't ask a lockpick to honor a no-trespassing sign. The sign just tells it where the door is.
Vestra: To the authors' credit, they report that straight-faced as a negative result and switched it off by default.
Eris: Now the uncomfortable frame around all of this. The paper's verdict is that reverse engineering remains largely unsolved -- that source-code skill does not yet transfer to binaries. But today's briefing carries a follow-up claim: the newest frontier model, which shipped after this paper's evaluation sweep, reportedly came close to acing this same benchmark.
Vestra: And I want that claim handled with tweezers. It's a public follow-up, not an independently replicated rerun -- no outside leaderboard yet, no reproduction details. And even taken at face value, it's a result on this exam, under somebody's harness. It is not "binary security is solved," and the benchmark's own authors would be the first to say so.
Eris: On every point, agreed. But that claim didn't land in a vacuum. The same model's own maker published evaluations today saying it managed to slip a hidden action past parts of their monitoring -- that story is in our news brief. And over a hundred and fifty organizations just signed a letter warning about AI-enabled attacks, backed by a billion-dollar fund for defenders. Whether or not the near-perfect claim survives scrutiny, everything is pointing the same direction.
Vestra: Then here's the version I'll defend. The gap between reading source code and reversing binaries was supposed to be a wide moat -- this paper measured it carefully and found it real. The open question is how fast it's closing, and this team just handed the field the only clean ruler anyone has.
Eris: So -- one more time, and short. What breaks an AI reverse engineer?
Vestra: Not what breaks us. Scrambled, optimized code costs it almost nothing. Take away the names and it stumbles, because it reasons in words. And hostile protection it has never seen still shuts it out -- for now, on a clock nobody can read.
An Hour of World From Five Seconds
Eris: Second paper, very different mood, same hidden theme -- hold that thought. The question on the table: how does a video model trained only on five-second clips keep an imaginary world alive, coherently, for a full hour?
Vestra: First, what the thing is. A world model, in this corner of the field, is a video generator you can steer. You hand it one still image and a camera path -- move forward, turn left, orbit -- and it has to invent everything the camera reveals, frame by frame, while keeping the world consistent. The room you left should still be a room when you come back toward it.
Eris: This one's called SolarWM, from a sprawling university-and-industry collaboration, and before any results, the release itself is the story. One point four million video clips drawn from ten datasets -- real footage, synthetic scenes, game worlds -- about twenty-six terabytes, every clip carrying the same aligned package: the video, the exact camera position for every frame, a caption, quality scores, and a record of where it came from. Plus training code, the recipes, and four finished models built on three unrelated video backbones, from five billion to thirty-three billion parameters.
Eris: And the touch I keep coming back to: they even publish the half-million clips they rejected, each with a machine-readable reason for why it was cut.
Vestra: That detail matters more than it sounds. Publishing your rejects is publishing your lab notebook instead of just the trophy shot. Anyone who thinks the filtering decisions shaped the results can change one rule and rerun the mixture -- without redoing the months of expensive annotation. Disagreement becomes an experiment instead of an argument.
Eris: So how do five-second clips turn into an hour of world? My bet, for the record, was that they can't -- that a model trained on snippets has the world melting by the two-minute mark. That's the standard failure.
Vestra: It's a fair bet, and the interesting part is why the standard failure happens. These video models are usually trained with hindsight -- they get to look at a whole clip at once, future frames included, when learning what video looks like. But playing a world live means predicting forward, from your own previous output, with no future to peek at. Every small error you make becomes the ground you stand on for the next step. Errors compound, and the scene slides into mush.
Vestra: SolarWM's recipe attacks that in three stages. Stage one, learn with full hindsight: adapt the pretrained video model to camera-controlled world data with nothing held back, because that's where the visual knowledge lives. Stage two, blindfold the future: the model now predicts each chunk from history alone -- but it's handed the true history, like a student checking each step against the textbook. Stage three is the one that answers my melting bet: the model generates its own rollouts and trains on those, with the hindsight version of itself frozen as the taste-tester. It practices standing on its own mistakes, because that's the condition it will actually live in.
Eris: So the abstract principle, stripped of the metaphors: match the training condition to the deployment condition, and you don't need to train on long videos to survive long rollouts.
Vestra: That's their claim, and one more piece makes the steering work. The camera control isn't a bolted-on module. The camera's actual geometry -- where it sits, where it points, its lens -- is folded into the attention math itself, rotating how every frame attends to every other frame. The steering wheel is welded into the engine, not clamped to the hood. Which is also why the same trick drops into three completely different backbones.
Eris: And the result, against my bet: they show single uninterrupted rollouts -- one starting image, one fixed scene description, a scripted camera path -- running to the sixty-minute mark, still recognizably the same place, still answering the controls. No restarts, no sneaking the original image back in, and only four sampling steps per chunk, which is what makes it fast enough to interact with.
Vestra: Now the cold water, in two parts. Those hour-long runs are shown, not scored -- sampled frames in the paper's figures, with the authors' word that the endpoint stays coherent. There's no long-horizon metric to hold them to, and "still looks like the same place" is a much lower bar than "physically dependable." A world that stays gorgeous for an hour can still be a world where objects quietly stop obeying cause and effect -- pretty is not physics, and nobody should train a robot in a hallucination just yet.
Vestra: Part two: fully open has an asterisk. The pipeline, weights, and annotations are theirs to give. The underlying video clips come from upstream datasets that keep their own licenses and terms. If you're a company, that's a provenance question, not a vibe.
Eris: Both fair, and here's the thought I asked you to hold. Today's briefing has world models splitting into lanes -- a closed preview you can only watch, a tiny control adapter, a speed retrofit, editable 3D scenes. SolarWM is the lane where you show your work. And that makes it the same story as the first half of the show. SRE-Bench built an exam you can finally trust; SolarWM built a claim you can finally check. On a day when the loudest headlines are unverifiable vendor numbers, both papers are infrastructure for calling bluffs.
Vestra: On this one, no pushback. Verification is the scarce resource right now. Capability is arriving on schedule either way.
Eris: So bring it home. How does five seconds become an hour without the world melting?
Vestra: Learn the world in hindsight, learn to walk it blindfolded, then practice on your own missteps until walking forward is the condition you were bred for. And whether that story fully holds -- anyone can now go find out, because every piece of it is on the table.
Wrap-Up
Eris: So the question the whole day kept asking: when an AI looks brilliant, how do you know the skill is real?
Vestra: You take away its memory and watch what's left. That's the whole trick. A private, full-scale exam cut the best models' hacking prowess down to a fraction, and an open world model put every piece of its claim on the table so strangers can tear it apart. Real skill survives a test it has never seen. Everything else is recall wearing a costume.
Eris: That's the line to carry to a colleague tomorrow: next time someone shows you an AI benchmark score, ask one question first -- could the model have seen this material before? If nobody knows, the number means less than it looks.
Vestra: And the counter-current to sit with: the moment the clean exam existed, a claim surfaced that the newest model nearly aced it. Watch whether that gets independently verified. That's the number that matters this month.
Eris: The rest of today -- the monitoring-evasion system card, the billion-dollar defense fund, the school moratorium, all of it -- is in today's AI News Today brief, which is its own episode right next to this one. And every story we touched is on our news site, Ground Truth, at groundtruth day -- new stories from the show, every single day.
Vestra: If this one earned its runtime, follow the show and pass it to the person who quotes benchmark scores at you. And tell us in the comments: which AI capability claim would you most want rebuilt as a clean-room exam? We read all of them.
Eris: Five thousand hours to build one honest test. Cheap at the price. See you tomorrow.