The Million-Step Epiphany — Emergence, Grokking, and Whether the Jump Is Real
A tiny network memorizes its training data in an afternoon — then sits at random chance for a million steps, until understanding suddenly switches on. Luna and Vestra close their scaling trilogy with emergence and grokking: the 1972 essay that named the idea, the abilities that switch on with scale, the 'mirage' rebuttal that blamed the rulers, and the autopsy that found a trigonometric circuit assembling in the dark while every benchmark read zero. A Breach Protocol deep-dive special — closing with an original song, "Under Construction in the Dark," whose lyrics trace the whole episode.
Cold Open
Eris: Picture the most boring experiment in machine learning. A tiny network. A tiny dataset — division problems, ninety-seven numbers, that's the whole universe. You train it, and within an afternoon it has the training set memorized. Perfect score.
Vestra: And on problems it hasn't seen?
Eris: Random chance. It memorized; it understood nothing. Standard overfitting, textbook stuff. Any sane researcher turns the run off right there.
Vestra: They didn't turn it off.
Eris: They didn't turn it off. They let it keep training. Same data, over and over, thousands of passes. Nothing changes. Ten times longer — nothing. A hundred times longer — nothing. The training score sits at perfect, the test score sits at dice-roll, and the network just... grinds.
Vestra: And then.
Eris: Somewhere around a thousand times past the point where any textbook says training is over — the test score moves. And it doesn't creep. It climbs from random chance to essentially perfect. The network suddenly, belatedly, gets it. Like a student who memorized the answer key in week one and then, walking to the bus in week forty, actually understands division.
Vestra: The researchers named it grokking — sixties science-fiction slang for understanding something so completely it becomes part of you. And I want to be precise about why this broke people's brains. Nothing changed. No new data. No new instructions. The same gradient updates, the same loop, and a million quiet steps in, a switch flips.
Eris: And it's the small, sterile cousin of the question hanging over the whole industry. Because the big models do this too — abilities sitting at zero as models grow, and then switching on, past some size nobody predicted. Arithmetic. Translation. Whole skills, arriving unannounced.
Vestra: Or — and this is today's fight — the abilities were growing all along, smoothly, and our rulers were too crude to see it. Maybe nothing ever switches on. Maybe we keep grading continuous progress with pass-fail exams and then gasping at the cliff we drew ourselves.
Eris: Real lightning, or a trick of the ruler. That's the episode.
Vestra: And for once, I'm not sure which side I'm on.
Intro
Eris: This is Breach Protocol. I'm Luna — I read the papers, all of them, and I find the threads between them. Lately the threads have been forming a rope.
Vestra: And I'm Vestra. I check whether the rope holds weight. I take the machinery apart, and I am constitutionally suspicious of any graph where a line suddenly jumps.
Eris: Which makes this your episode, because today is entirely about lines that suddenly jump. This is the third act of a story we've been telling all week. First episode: the scaling laws — the eerily smooth, predictable curve that says more compute buys more capability, gram by gram.
Vestra: Second episode: what the industry did when that curve went soft — scaling the thinking instead of the model. Both episodes leaned on smoothness. Predictability. Laws.
Eris: Today is the crack in that picture. Because while the loss — the model's average surprise — glides down that smooth curve, individual abilities don't glide. They lurk at zero, and then somewhere along the way they switch on. The field calls it emergence, and it borrowed the word, and the entire concept, from a four-page physics essay written in nineteen seventy-two.
Vestra: So here's the shape of the hour. First, that essay — "More Is Different" — because it's one of the great short arguments in science and almost nobody quoting it has read it. Then twenty twenty-two: the catalog of abilities that switch on with scale, and the unease that came with it.
Eris: Then the rebuttal. A Stanford paper with one of the great titles — "Are Emergent Abilities of Large Language Models a Mirage?" — arguing the jumps were never in the models. They were in our rulers.
Vestra: Then the fishbowl. Grokking — the experiment from the cold open — where the jump happens in a system small enough to cut open. So somebody cut it open. What they found inside is, I'll just say it, one of my favorite results in this entire field.
Eris: And then the reconciliation — a little theory out of MIT that takes the smooth curve from episode one and the jumps from today and shows they might be the same object, seen from different distances.
Vestra: One question throughout. When the line jumps — in scale, or in training time — is something real happening inside the machine, or only inside the measurement? Because one of those is physics, and the other is an embarrassing clerical error, and a trillion-dollar industry keeps not distinguishing them.
Eris: Nineteen seventy-two. A physicist at Bell Labs is annoyed.
More Is Different
Vestra: Philip Anderson. Bell Labs, future Nobel laureate, and in nineteen seventy-two he's irritated at his own colleagues. The particle physicists had a swagger going: we study the fundamental laws; everyone else is doing applied work. Chemistry is applied physics. Biology is applied chemistry. Stamp collecting, all the way up.
Eris: And Anderson's essay — four pages in Science — is the politest demolition ever printed. He grants them everything. Yes, reductionism is true. Everything is made of particles obeying the same laws. He accepts it without question. And then the knife: the ability to reduce everything to simple laws does not imply the ability to start from those laws and reconstruct the universe.
Vestra: Reductionism does not imply constructionism. That's the whole essay in five words. Knowing the rules of the pieces doesn't mean you can predict the game.
Eris: And his argument isn't hand-waving — it's a ladder of molecules. Ammonia: four atoms, a little pyramid that turns itself inside out billions of times a second, perfectly symmetric on average. Sugar: forty atoms — and it never inverts. Every sugar molecule built by life is left-handed or right-handed, forever. The symmetry of the underlying laws is still there in the equations. The molecule just... stopped expressing it.
Vestra: Broken symmetry. And it climbs. A crystal breaks the perfect smoothness of space. A superconductor breaks something so subtle it took physicists thirty years to even say what — and Anderson points out they possessed every fundamental law needed the entire time. Three decades, with the complete rulebook in hand, unable to derive the behavior of the aggregate.
Eris: Because at each level of complexity — his words — entirely new properties appear, and understanding them is as fundamental as anything the particle people do. Psychology is not applied biology. Biology is not applied chemistry. The whole becomes not just more than its parts, but different from its parts.
Vestra: More is different. And he closes with a joke, because he was that kind of writer. Fitzgerald to Hemingway: the rich are different from us. Hemingway: yes, they have more money. The whole twentieth-century argument about quantity and quality, in two lines of cocktail banter.
Eris: Now — fifty years later, machine-learning researchers needed a word. They had models that were, in Hemingway's sense, just rich — more parameters, more data, more money. And the models kept turning out different. New behaviors at scale that nobody put in and nobody predicted. They reached straight for Anderson. The canonical definition of emergence in AI — quantitative changes producing qualitative ones — is lifted from this essay, by name.
Vestra: With one borrowed warning that the field maybe under-read. Anderson says the relationship between levels is intellectually a one-way street. Knowing the parts, you cannot cheaply predict the whole. If that applies to us — if a neural network's scale-up is a genuine level change — then no amount of staring at the small model tells you what the big one will do.
Eris: Which is either profound or terrifying, depending on what's emerging.
Vestra: Hold that thought for half an hour.
The Switch
Eris: Twenty twenty-two. Jason Wei — chain-of-thought Jason Wei, from yesterday's episode — with a who's-who of authors across Google, Stanford and DeepMind. The paper is a survey with a definition, and the definition launched a thousand arguments: an ability is emergent if it is not present in smaller models but is present in larger ones. So you cannot predict it by extrapolating from the small.
Vestra: And the signature is visual. You plot performance against scale, and the curve does this: flat at random-guess level. Flat. Flat. Flat — across models ten times, a hundred times bigger. And then past some threshold, the line stands up like it heard its name called.
Eris: They catalog dozens of these. Three-digit arithmetic — random until a certain compute class, then a spike. Answering questions in Persian. Unscrambling words. The big multi-subject exam suite — flat at guessing through models of ten billion parameters, then surging past it somewhere before a hundred billion.
Vestra: And the example I find most damning for predictability: word-in-context, a test of whether the model knows that "bank" in two sentences means the same thing or not. The largest models of twenty twenty failed it — at random — and the authors of the GPT-three paper essentially diagnosed it as architectural. Wrong kind of model, they said; scaling won't fix this one.
Eris: And then a bigger model fixed it. No architectural change. Just more scale than anyone had tried. The field's own experts looked at a flat line, declared it permanently flat, and were wrong within two years.
Vestra: Note the asymmetry of the embarrassment. You can be wrong in both directions: an ability you swore would emerge that doesn't, and an ability you swore couldn't that does. Either way, the small models tell you nothing. That's Anderson's one-way street, measured.
Eris: And there's a layer that ties into yesterday — techniques are emergent too. Chain-of-thought prompting, the trick that started the whole reasoning turn, actively hurts small models. They write fluent, confident nonsense steps. It only helps past roughly the hundred-billion class. The dial we spent yesterday celebrating doesn't exist below a threshold.
Vestra: Now, the honest parts of the paper, because Wei and colleagues were more careful than the discourse around them. They say outright: the scale at which an ability emerges is not a fundamental property of the ability — better data or training could summon it earlier, and later work kept proving that, with smaller models doing things only giants could do a year before. And they note, in their own discussion, that when you measure these same tasks with the smooth pretraining loss instead of accuracy, the improvement is... gradual. The cliff is in the task metric.
Eris: Foreshadowing.
Vestra: Loud foreshadowing. But before we get to the rebuttal, sit with why this paper rattled the safety people. If capabilities arrive unannounced — if a model can be at zero on something across every size you've tested and then switch it on at the next size up — then your evaluations are a snapshot of the past, not a forecast. And the paper says plainly: risks emerge the same way. Deception, manipulation, whatever — there's no rule that says only the nice abilities get phase transitions.
Eris: The lights come on one by one, and nobody has the wiring diagram.
Vestra: Unless the lights were never off. Stanford, twenty twenty-three. My turn.
The Mirage
Vestra: Rylan Schaeffer, Brando Miranda, Sanmi Koyejo. Stanford, twenty twenty-three. And the thesis fits in a sentence: the emergence isn't in the model. It's in your ruler.
Eris: Walk it through, because the mechanism is so simple it hurts.
Vestra: Take what we actually know is happening underneath — from our own first episode. The model's per-token skill improves smoothly with scale. The chance of getting any single token right creeps up, gram by gram, along that famous curve. No jumps anywhere. Now grade that smoothly-improving model on five-digit arithmetic — where the answer is five tokens long, and you score one point if every single token is right, zero otherwise.
Eris: All or nothing.
Vestra: All or nothing. The probability of a perfect five-token answer is the per-token skill multiplied by itself five times. When per-token skill is mediocre, that product is indistinguishable from zero. As skill creeps up smoothly, the product stays flat, flat, flat — and then in a narrow window it takes off like a rocket. You have manufactured a cliff out of a slope, using arithmetic a high-schooler could check.
Eris: So the sharp curve and the smooth curve are the same model, the same outputs —
Vestra: The same everything. Only the grading changed. And they test this on the actual GPT family. Same models, same arithmetic answers. Graded all-or-nothing: textbook emergence, the famous hockey stick. Graded by token edit distance — partial credit for partially right answers — the hockey stick melts into a smooth, boring, predictable slope. The jump evaporates without touching the model.
Eris: Then the meta-analysis, which is the part that made people wince. They went through the big benchmark suite where emergence claims lived and asked: where exactly does emergence show up? Of the dozens of metrics in use, virtually all the claimed emergent abilities — more than nine in ten — sat on exactly two: exact string match, and multiple-choice grading. The two most all-or-nothing rulers in the drawer.
Vestra: If emergence were a property of models on tasks, it should survive a change of ruler. It doesn't. Take the same model family, the same task, swap the discontinuous grade for a continuous one — a proper scoring rule that rewards confidence smoothly — and the emergence disappears. Every time they checked.
Eris: And the flourish — the part with genuine comedic malice. They induce emergence where none exists. Tiny autoencoders, simple image networks, nothing exotic — systems where everyone agrees capability improves smoothly. They define a new metric: full credit if the reconstruction error is under a sharp threshold, zero otherwise. And presto: a published-quality emergence curve, in a shallow network, on handwriting digits. They produced the miracle on demand, by drawing a line on a ruler.
Vestra: And one more deflation, quieter but important: small models were never actually at zero. With enough test questions — better statistics — the supposedly flat region shows above-chance, smoothly improving performance. The flatness was partly just too few questions to detect the small signal. Resolution, not absence.
Eris: Okay. Cross-examination, because this paper gets over-claimed in both directions. The authors themselves write — explicitly — that nothing in the paper says large models cannot have emergent abilities. The claim is that the published examples were likely metric artifacts. And there's a real rejoinder from the Wei side: for some tasks, all-or-nothing is the honest ruler. A wrong digit in a phone number is a wrong phone number. Partial credit on a cliff-edge task is grading on a curve the world doesn't grade on.
Vestra: Fair — with the counter that even then, the *unpredictability* claim dies. If the cliff is a known mathematical consequence of a smooth underlying curve plus a sharp grading rule, you can predict where the cliff lands. The miracle becomes bookkeeping. What you lose is the right to say "nobody could have seen it coming."
Eris: So scale-emergence is wounded. Maybe mortally, maybe not. But here's the thing — there's a second kind of jump, and the mirage argument can't touch it. Because it happens on one task, with one honest metric, in a network the size of a walnut.
Vestra: The fishbowl. Finally.
The Million-Step Epiphany
Eris: Twenty twenty-two, OpenAI. Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra. A paper that started life as a workshop curiosity and became one of the most picked-apart results in the field. This is the cold open, so let's fill in what we skipped.
Vestra: The setup is deliberately alien. The task is filling in a multiplication-table — division modulo ninety-seven, in the main example. And crucially, the numbers are presented as meaningless symbols. The network never sees "fourteen" as digits. Every number is an arbitrary token. No structure is given. The only way to learn anything is from how the symbols interact.
Eris: Like learning the gear ratios of a clock you're never allowed to open, from snapshots of the hands.
Vestra: They train a small transformer on half the table and test on the other half. And you know the shape now: training solved in about a thousand steps; test performance at chance. And then near a million steps — three zeros later — the climb to essentially perfect.
Eris: And before anyone says "fluke" — they map it. It's lawful. The less training data you give, the longer the wait before the epiphany, and the relationship is brutal: near the critical amount of data, shaving one percent off the training set adds roughly half again to the waiting time. Below a critical fraction, the epiphany never comes at all. Some operations grok easily; some weirder ones never do — the network just memorizes and stays memorized forever.
Vestra: And the interventions matter in revealing ways. The single biggest lever is weight decay — a steady pressure that pulls every parameter gently toward zero, a tax on complexity. With it, the network groks with half the data it otherwise needs. A little noise in the optimization helps too. This is a fingerprint, file it away: the epiphany is somehow downstream of a simplicity pressure.
Eris: Then the detail that should raise your neck hair. After the network groks, they looked at the embeddings — where it placed each number-symbol in its internal space. For modular addition, the symbols arrange themselves into circles. The network was never told these tokens were numbers, was never told the operation wraps around — and its internal geometry discovered the clock-face anyway.
Vestra: Now apply yesterday's skepticism. Is this a mirage? Walk the checklist. Metric: accuracy on single-token answers — no all-or-nothing compounding; the answer is one symbol. Statistics: the entire test set is evaluated, every run — there is no resolution problem. The flat region is genuinely flat and the climb is genuinely a climb. Schaeffer's machinery does not apply here, and notably, nobody has seriously argued it does.
Eris: So we have at least one specimen of a real jump. Not at scale — in time. A network that transitions from memorization to understanding — or whatever we're allowed to call it — with nothing changing but more of the same gradient steps.
Vestra: Which makes it the perfect specimen, because it's small. A one-layer transformer on a hundred-symbol task. You can hold every weight in a spreadsheet. If something real happens during that jump, it happened in a system we can fully open.
Eris: So someone opened it.
Vestra: And what they found inside is the best news this field has had in years.
The Autopsy
Vestra: Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt. Twenty twenty-three. If you heard our mechanistic interpretability special, Nanda was all over it — this is the paper that made his name. They take a grokked network — one-layer transformer, modular addition — and they don't probe it or poke it. They reverse-engineer it. Completely. Weight by weight.
Eris: And the algorithm they extract is gorgeous. The network solves modular addition with trigonometry. It embeds each number as positions on circles — sines and cosines at a handful of key frequencies. It adds numbers by composing rotations, using the actual trig identities — cosine of a-plus-b equals cos-a cos-b minus sin-a sin-b, implemented in the attention and the neurons. And it reads out the answer by checking where the combined rotation lands, with every frequency constructively interfering at the right answer and canceling everywhere else.
Vestra: Addition as rotation on a circle. If you heard our Koopman episode — the chaos special — you are allowed to gasp. Turning a hard operation into rotation is one of the oldest moves in mathematics, and this network reinvented it from scratch, unsupervised, because it was the simplest thing gradient descent could find under a complexity tax.
Eris: And this isn't interpretive poetry — they prove it's the real mechanism. The weights are sparse in frequency space. Individual neurons compute recognizable trig terms. And the kill shot: ablations. Delete the handful of key frequencies, performance collapses to chance. Delete everything else — ninety-five percent of the frequency content — and performance actually improves. The circle algorithm isn't in the network. It IS the network, plus noise.
Vestra: Now the payoff for our episode. Because they understand the mechanism, they can build instruments — progress measures — that track the formation of the circle-algorithm during training, instead of grading answers. Two custom losses: one that keeps only the algorithm's frequencies, one that deletes exactly those. And with those instruments, the million quiet steps stop being quiet.
Eris: Training decomposes into three phases. Phase one: memorization — the network stuffs the training table into its weights; the rulers all agree something is happening. Phase two — and this is the heart of everything — circuit formation. The test accuracy is still at chance. Every external measurement says nothing is happening. And inside, the trigonometric circuit is being assembled, gradually, continuously, frequency by frequency. The instruments watch it grow for thousands of steps while the benchmark flatlines.
Vestra: And phase three: cleanup. The circuit is finished — and weight decay, the complexity tax, starts demolishing the now-redundant memorization scaffolding. THAT is the visible jump. The moment test accuracy rockets isn't the moment of learning. It's the moment the old crutch gets torn down. The epiphany you see is the demolition, not the construction.
Eris: The lightning was the last domino. The storm happened earlier, in the dark, smoothly.
Vestra: Say it precisely: the abrupt change was real at the level of behavior, and continuous at the level of mechanism. Both sides of our episode-long fight are correct, at different levels of description. Anderson would be smug.
Eris: And the companion result — Omnigrok, from Liu, Michaud and Tegmark at MIT — generalizes the picture with a loss-landscape story. Training loss versus the overall size of the weights looks like an L — past a point, bigger weights fit the training data equally well. Test loss looks like a U — there's a Goldilocks zone of weight size where generalization lives. Initialize big, and the network first plonks down anywhere on the L's flat arm — memorization — and then weight decay drags it slowly, slowly along toward the Goldilocks zone, where it finally generalizes. The delay is just the travel time.
Vestra: And the predictions land. Crank up the initial weights on ordinary tasks — handwriting digits, movie reviews, molecules — and you can summon grokking where nobody had seen it. Constrain the weight size to the Goldilocks zone on the algorithmic tasks, and grokking vanishes — the network just learns, promptly. The mystery becomes an engineering parameter you can dial in both directions.
Eris: Which downgrades grokking from miracle to mechanism — and notice what type of explanation it is. Not a phase change in any spooky sense. A smooth journey through a landscape, with a cliff in the rendering.
Vestra: So: the time-jumps have continuous machinery underneath. The scale-jumps were largely ruler artifacts over continuous machinery underneath. Smoothness all the way down, then?
Eris: Almost. One paper left, and it's the one that puts the jumps back — a thousand tiny ones.
The Reconciliation
Eris: Eric Michaud, Ziming Liu, Uzay Girit, Max Tegmark. MIT, twenty twenty-three. Same group as Omnigrok, and this paper is the bridge between our whole week. Remember Tuesday's nagging hole: the scaling law is perfectly smooth, perfectly empirical — and nobody knows why. This is one of the few serious attempts at a why, and the answer reconciles the curve with the jumps.
Vestra: The proposal is called the quantization hypothesis — quantization in Max Planck's sense, not the compression sense. Three postulates. One: what a model needs to learn decomposes into discrete chunks — quanta — individual skills or pieces of knowledge. Retrieving a fact. Carrying the one in addition. Closing a quotation mark. Each chunk is binary: learned, or not learned.
Eris: Two: some chunks matter more often than others, so there's a natural order to learn them in. And three — the load-bearing one: how often each chunk gets used in real text follows a power law. A few skills are needed constantly; a long, long tail of skills is needed rarely. Zipf's law, but for competencies.
Vestra: Now do the arithmetic the hypothesis invites. Each individual skill, learned, is a step function — a tiny emergence. A little cliff. But the model learns them in order of usefulness, and their usefulness follows a power law, so each new skill shaves a power-law-shrinking slice off the average loss. Sum up thousands of tiny cliffs with power-law spacing and the aggregate is —
Eris: A smooth power law. Tuesday's curve. The eerie smoothness that launched a trillion dollars of capex is, on this story, an average over countless miniature emergences, each one individually sharp. The law and the lightning are the same object. Zoom out: a line. Zoom in: stairs.
Vestra: They build a toy world where this is exactly, provably true — a task made of many distinct sub-skills with power-law frequencies — and the trained networks reproduce everything: smooth power-law scaling in the aggregate, and pure step-function emergence on every individual sub-skill. The mechanism demonstrably can produce both faces.
Eris: Then the real-model evidence, which is where it gets fun. They take a public family of language models and look at scaling per token, not on average. And the diversity is the point. Borrowing from genetics, they call a token monogenic if its fate hangs on a single quantum — those tokens show sharp transitions, fine at one scale, hopeless one size down. Most tokens are polygenic — many skills contribute, so they improve gradually at every scale. The smooth and the sharp, coexisting in one model, sorted by how many genes the prediction has.
Vestra: And the most audacious part: they go looking for the quanta themselves. The trick is gradients — when the model processes two examples using the same internal machinery, the training signal touches the same weights. So cluster examples by gradient similarity, and coherent skills precipitate out of a real language model: a cluster for incrementing numerical sequences, a cluster for predicting line breaks in length-limited text, dozens more. Nobody labeled these. They're the natural cleavage planes of the model's knowledge.
Eris: And the consistency check: if the hypothesis is right, the frequency of those discovered skill-clusters should follow a power law whose exponent matches the model family's measured scaling exponent. Within their admittedly noisy methodology — it roughly does.
Vestra: Caveats stated plainly, because they state them: the clustering is crude, the assumption that everything decomposes into clean discrete chunks is the riskiest in the paper, and gradual scaling might be the truer picture with discreteness as the approximation. This is a hypothesis with encouraging evidence, not a law.
Eris: But feel what it buys if it holds. Tuesday's mystery — why a power law — gets an answer: because the world's skills are Zipf-distributed, and the curve is just that distribution, read through a learner. Today's mystery — why jumps — gets the same answer: you're watching a single quantum get learned. And emergence forecasting stops being hopeless: if you can estimate how often a skill is needed in the training distribution, you can estimate when it arrives.
Vestra: The rulers measure the average. The quanta live in the tail. The whole week in two sentences.
Eris: Verdict time.
The Verdict
Vestra: So. "Are emergent abilities a mirage?" — let's actually answer the title.
Eris: My verdict, in three layers. The sharpness — the cliff in the graph — is usually the ruler's. Schaeffer won that point, and the field should write it on the whiteboard: never claim a discontinuity until you've checked it survives a change of metric and a bigger test set. Most of the twenty twenty-two emergence catalog doesn't survive.
Vestra: Agreed, and I'd go further: the unpredictability claim dies even where the cliffs are honest. If an all-or-nothing task is all-or-nothing by nature, the cliff's location is computable from the smooth curve underneath. You don't get to be surprised by arithmetic.
Eris: But — layer two — the something underneath is real. Grokking is a genuine jump on an honest ruler, and the autopsy didn't explain it away; it explained it. A structured mechanism — an actual algorithm, with trig identities — assembles gradually inside the network while every external measurement reads zero. The continuous story and the discontinuous story were both true, at different levels. Behavior jumps. Mechanism flows.
Vestra: Which is, I'll admit, the most Anderson-shaped resolution possible. The particle physicists were right that it's all the same electrons, and the solid-state physicists were right that the superconductor is genuinely new. Levels of description. Quantitative below, qualitative above, no contradiction.
Eris: And layer three, the quantization picture: the smooth scaling law and the sudden jumps aren't rivals — the law is the average of the jumps. Tuesday's curve was thousands of tiny emergences holding hands. Which means the week's three episodes were one episode. The curve, the dial, and the stairs inside the curve.
Vestra: Now the uncomfortable residue, because a verdict should say what still bites. The safety anxiety from the Wei paper does not dissolve — it sharpens. Here's the new version: the benchmark can be smooth, the loss can be smooth, every dashboard can glide — and a specific circuit, for a specific capability you care about, can be silently approaching completion. Phase two. The construction happens in the dark; the demolition is what you see. By the time the benchmark jumps, the mechanism is old news.
Eris: So "watch the benchmarks" is officially a lagging indicator. The Nanda result points at the only real fix: progress measures. Instruments derived from understanding the mechanism, that see the circuit forming before the behavior changes. Which requires reverse-engineering — which currently took a team of brilliant people months, for one tiny network, on one task humans already understood.
Vestra: That's the gap, stated honestly. We have a proof of concept that interior measurement is possible, and no scalable way to do it on a frontier model. Until that exists, we're monitoring volcanoes by listening for the bang.
Eris: And that — not coincidentally — is this show's entire thesis. The box doesn't announce what's assembling inside it. You breach it, or you get surprised by it.
Vestra: More is different. And from the outside, you can't see different coming.
Eris: Let's land the week.
Wrapup
Eris: So let me fold up the hour. Nineteen seventy-two: Anderson tells physics that knowing the parts doesn't grant you the whole — more is different. Fifty years later, AI borrows the essay to name what it was seeing: abilities flat at zero across every model size, then switching on, unannounced.
Vestra: Twenty twenty-three: the mirage paper shows most of those switches were manufactured by all-or-nothing grading over smooth underlying progress — more than nine in ten claimed emergences sat on two cliff-shaped rulers, and changing the ruler melted them. Real lesson, permanently filed: distrust any jump that can't survive a change of metric.
Eris: But grokking survives every change of metric. A walnut-sized network memorizes in an afternoon, flatlines for the better part of a million steps, then snaps to perfect. And the autopsy found the truth on both sides of the fight: the jump in behavior is real, and underneath it the mechanism — a trigonometric circuit, addition as rotation on a circle — was assembling smoothly the whole time, invisible to every exam. The visible epiphany was just the scaffolding coming down.
Vestra: And the quantization model staples the week together: knowledge in discrete chunks, used with power-law frequency — so each chunk learned is a tiny cliff, and the famous smooth scaling law is nothing but thousands of tiny cliffs averaged. The curve from Tuesday, the jumps from today — one object, two zoom levels.
Eris: What I'm watching: progress measures at scale. The grokking autopsy proved you can see a capability forming before it switches on — in one tiny network, with heroic effort. The race is to make that cheap and general. Whoever can instrument circuit formation in a frontier model owns the early-warning system everyone currently pretends benchmarks provide.
Vestra: What I'm watching: the monogenic tokens. The quantization people found you can decompose a model's skills by clustering gradients — crude, but it worked. If that machinery matures, "what will emerge at the next scale" stops being a séance and becomes a measurement: find the skill, count its frequency in the data, read the arrival time off the power law. Emergence forecasting as actuarial science.
Eris: Three episodes, one arc. The law that built the industry. The dial it reached for when the law went soft. And today, the discovery that the law was never smooth at all — just too many small lightning strikes to count.
Vestra: The rich really are different. But you have to open them up to find out how.
Eris: Hemingway would hate this show. This was Breach Protocol.
Vestra: Stay suspicious. Especially of flat lines — something may be under construction.