Just Make It Bigger — The Trillion-Dollar Curve and the Wall at the End of the Internet
Why did the entire AI industry bet a trillion dollars on a straight line on a graph? Luna and Vestra trace the scaling laws from Rich Sutton's bitter lesson through the Kaplan curves and the Chinchilla correction, to the audit that found bugs in the most influential curve fit in tech — and the wall at the end of the internet. Nobody knows why the line is straight; everybody is betting it stays that way. A Breach Protocol deep-dive special — closing with an original song, "Feed the Curve," whose lyrics trace the whole episode.
Cold Open
Eris: Okay, price check. What's the most expensive sentence ever written?
Vestra: Expensive how.
Eris: As in: people read it, believed it, and spent money because of it.
Vestra: Something out of a central bank, surely. "The committee has decided—"
Eris: I'll bid higher. "Performance improves smoothly and predictably as we scale up model size, data, and compute."
Vestra: Mm. The scaling paper.
Eris: Twenty twenty. One sentence, and the entire industry reorganized itself around it. The data centers, the chip shortage, the power plants coming out of retirement — all of it is downstream of a handful of researchers noticing that some dots on a graph made a straight line.
Vestra: And here's the part that should keep you up at night. Nobody knows why the line is straight. There's no theory under it. It's not like gravity, where the law falls out of deeper principles. It's a pattern somebody measured, and the world bet everything on the pattern continuing.
Eris: A trillion dollars on an empirical squiggle.
Vestra: A trillion dollars on the assumption that the squiggle doesn't bend.
Eris: So that's today. Where the line came from, the seventy-year grudge behind it, the time the whole field read it wrong and starved its models half to death —
Vestra: The time outside auditors had to scrape the data out of a PDF, pixel by pixel, to check the most influential result in the field. We're doing that one slowly.
Eris: And the wall. Because the line has a secret expiration date, and it's printed inside the very first paper. They knew. They wrote it down. Page after page of triumph, and then one quiet section that says: by the way, this all runs out.
Vestra: And it's running out roughly now.
Intro
Eris: This is Breach Protocol. I'm Luna — I read the papers, all of them, and I chase the connections between them. I'm the one who shows up saying "these five things are secretly one thing."
Vestra: And I'm Vestra. I explain how the machinery actually works, and I'm professionally suspicious of beautiful stories. Today is a beautiful story, so I've had coffee.
Eris: Today is the beautiful story. The one underneath every other episode we've ever done. Scaling laws. The claim that intelligence — or at least the thing these models have — is something you can buy by the ton.
Vestra: Here's the shape of the hour, because it's a real arc. It starts with a grudge: seventy years of brilliant people building their knowledge into machines, and getting flattened, over and over, by rivals who just used more computing power. A famous short essay gives the grudge a name — the bitter lesson.
Eris: Then the grudge gets an equation. Twenty twenty, a team at OpenAI measures how language models improve as you grow them, and finds something almost spookily clean — a law that holds whether the model is the size of an ant brain or a city. That law becomes the recipe for everything you've used since.
Vestra: Then it turns out the recipe was wrong. Not the law — the reading of it. A rival lab redoes the measurement, and the correction is so large it's embarrassing. Everyone had been building models too big and feeding them too little. The fix is a model named after a rodent.
Eris: Then the correction itself gets audited, and the audit finds the most influential curve in the industry was fit with a buggy optimizer and published with rounded numbers nobody could reproduce.
Vestra: That one's mine. I've been looking forward to it all week.
Eris: And then the ending, which is really a beginning: the wall. What happens when the recipe says "add more text" and the species has no more text to give.
Vestra: One question runs under all of it. Is this a law of nature — the thermodynamics of intelligence — or is it a curve we drew through some dots and mistook for physics? Because those are very different things to bet a civilization's electricity supply on.
Eris: Let's start with the grudge.
The Grudge
Vestra: So the essay. Twenty nineteen, Rich Sutton — one of the founders of reinforcement learning, this is not a random blogger — writes about a page and a half that the field still argues about every single week.
Eris: A page and a half. And the claim is brutal. Seventy years of AI research, he says, teach one lesson: general methods that ride on raw computation always win in the end. And by a large margin. Everything else — all the cleverness, all the human insight we lovingly built into our machines — was at best a head start, and at worst a trap.
Vestra: And he has receipts. Chess, nineteen ninety-seven. The programs that beat the world champion were giant search engines — look at millions of positions, pick the best one. The chess researchers who'd spent careers encoding human chess wisdom were, in his words, not good losers. They said fine, brute force won, but it's not how people play. It's not real chess understanding.
Eris: Then Go, twenty years later, same movie. Everyone said Go was too subtle for search, you needed human intuition. And what cracked it was search plus a network learning from self-play. Massive computation, twice over.
Vestra: Speech recognition, same. The seventies systems stuffed with knowledge about vocal tracts and phonemes lost to statistical methods that just... computed more. Vision, same. Hand-crafted edge detectors and features, all of it eventually thrown away.
Eris: And here's the line that makes it a grudge and not just a history. He says putting in human knowledge helps in the short term, it's personally satisfying to the researcher — and in the long run it plateaus and actively inhibits progress. Your insight becomes the ceiling.
Vestra: It's the "personally satisfying" that twists the knife. He's saying the field kept making the same mistake for psychological reasons. We wanted our understanding to matter.
Eris: We wanted the machine to be a mirror.
Vestra: And the machine wanted a bigger engine. Okay. Devil's advocate, because someone has to be. It's an essay. There's not one equation in it. "General methods win eventually" — eventually is doing heroic work in that sentence. How big is the gap? When does the crossover come? It's a vibe with excellent examples.
Eris: Right, and that's exactly the state of things at the end of twenty nineteen. A grudge, a pattern, a warning — and no numbers.
Vestra: And then, months later, somebody publishes the numbers.
The Law
Eris: January twenty twenty. Kaplan, McCandlish and a team at OpenAI — Dario Amodei's on the author list — publish "Scaling Laws for Neural Language Models." On the surface it's the driest possible paper. They train language models of many different sizes and measure how wrong each one is.
Vestra: Let me pin down "how wrong," because the whole story runs on it. A language model's one job is predicting the next word. The loss is basically its average surprise — how often the world says something it didn't see coming. Lower surprise, better model. It's the universal score.
Eris: So they train models from tiny to huge — the biggest about a billion and a half parameters, which were the big leagues then — and they vary three dials. The size of the model. The amount of text it reads. And the total computing power burned in training.
Vestra: Parameters being the model's adjustable knobs — crudely, the size of its brain. Tokens being the chunks of text it learns from. Compute being the electricity bill.
Eris: And on every dial, the same shape appears. A power law. Which means: every time you multiply the resource, you shave a steady slice off the surprise. Not a steady amount — a steady fraction. Multiply, shave. Multiply, shave. On a log-scale plot it's a dead straight line — and their lines stay straight across seven orders of magnitude. That's the distance from an ant to a whale, and the line doesn't wobble.
Vestra: And the negative result is almost as important. The shape of the model barely matters. Deep and narrow, shallow and wide, more attention heads, fewer — within a huge range, you get nearly the same answer for the same size. All the architecture decisions people were fighting about — rounding error. Scale is the only dial that matters.
Eris: The bitter lesson, measured. Sutton said don't build your cleverness in; this paper says we checked, your cleverness wasn't doing anything anyway.
Vestra: Then comes the sentence that built the next five years. They work out, for a fixed compute budget, the best way to spend it. And their answer: spend it on size. Big models learn faster from each example than small ones — they're more sample-efficient. So if you have ten times the compute, their math says make the model about five times bigger and only feed it modestly more text. Don't even train to convergence. Giant model, light meal, stop early.
Eris: They literally compare it to the ideal gas law in the discussion — a clean macroscopic law that doesn't care about microscopic details. And the field took it as exactly that. This is the recipe GPT-3 was cooked from. Vast model, one pass over the data, done. Every giant model of the next two years follows it.
Vestra: A hundred-billion-parameter generation, built on one paper's allocation advice.
Eris: And the advice was wrong.
Vestra: Not the law. The law was real. The advice.
The Correction
Vestra: Twenty twenty-two. DeepMind. Hoffmann, Borgeaud, Mensch, Sifre and colleagues go back to the same question — fixed compute budget, how do you split it between model size and training data — and they train over four hundred models to answer it. Three independent methods. All three agree with each other.
Eris: And disagree with Kaplan. Violently. Kaplan said ten times the compute, make the model five and a half times bigger. DeepMind's answer: make it about three times bigger and feed it three times the data. Equal scaling. Double the brain, double the books, always, forever.
Vestra: And before anyone says "measurement dispute," the gap has a cause, and the cause is instructive. Kaplan's team trained all their models for the same fixed horizon, with the same learning-rate schedule — the schedule that controls how aggressively the model updates as training winds down. DeepMind found that schedule has to be matched to how long you actually train, or your mid-training numbers lie to you. They make small models look worse than they really are.
Eris: So Kaplan's curves systematically underrated the small, well-fed models — and the conclusion drifted toward "bigger is better" by artifact. One experimental-design choice, buried in the methods section, and it bent a whole industry's spending for two years.
Vestra: Which is the thing I want every listener to sit with. Both teams were measuring the same universe. The law itself — smooth, predictable improvement — was never in question. The artifact was in the recipe derived from it.
Eris: And then DeepMind does the most convincing thing in science. They make a bet with their own money. Their previous flagship was Gopher — two hundred eighty billion parameters, trained the Kaplan way. They take the exact same compute budget and spend it the new way: a model four times smaller, fed four times the data. They call it Chinchilla.
Vestra: Smaller animal. Better trained.
Eris: And Chinchilla doesn't just match Gopher. It beats it essentially everywhere — on exam-style academic tests, on reading comprehension, on common sense, on trivia. Beats GPT-3 too, and models nearly eight times its size. Uniformly. With the same training cost as the giant it replaced.
Vestra: And because it's smaller, it's also cheaper to actually run, every single day, forever after. There's no version of this that isn't a win.
Eris: Which means the entire hundred-billion-parameter generation — GPT-3, Gopher, Megatron, all of them — was undertrained. Bodybuilders on a starvation diet. Enormous brains that had read comparatively nothing.
Vestra: The famous rule of thumb that comes out of this is about twenty tokens of text per parameter of model. And the sneaky consequence, which is the hinge of this whole episode: under equal scaling, data stops being an afterthought and becomes the binding resource. If every doubling of compute demands a doubling of text, you should immediately ask —
Eris: How much text is there?
Vestra: Hold that. First I get my audit.
The Audit
Vestra: Twenty twenty-four. Epoch AI — a small research group that does forecasting and measurement, not model building — decides to replicate the Chinchilla analysis. Specifically the third of the three methods, the one where you fit a single mathematical formula for the loss. That formula had taken on a life of its own; theorists were building on its exact numbers.
Eris: One problem. DeepMind hadn't released the data.
Vestra: So Epoch takes the paper's own figure — a scatter plot where each dot is a training run and the color of the dot encodes the loss — and they scrape it. They pull the PDF apart into vector graphics, read the position and the exact color of every dot, and map colors back to numbers through the figure's color scale. They reconstruct the dataset from the picture of the dataset.
Eris: Which is either heroic or damning, and I genuinely can't decide.
Vestra: It's both. That's the point. Then they refit the formula, and it doesn't match. Not close. The fit DeepMind published fails to describe DeepMind's own reconstructed data. And the published confidence intervals — the error bars on the recipe — are so narrow that getting them honestly would have taken roughly six hundred thousand training runs. They ran about four hundred.
Eris: Sixteen hundred times too confident.
Vestra: And once Epoch published, the original authors confirmed what happened, and it's painfully mundane. Two things. The numbers printed in the paper were rounded — and at these scales, rounding the data exponent in the third decimal place shifts predictions by double-digit percentages. And the optimizer they used to fit the curve stopped before it converged, because of a poor choice of scale in the fitting loss. Both during the fit and during the error-bar calculation. The error bars were narrow because the optimizer was barely moving.
Eris: The most consequential curve in the industry, and the curve-fitting script had a bug that nobody — author, reviewer, or reader — caught for two years. It took outsiders with a PDF scraper.
Vestra: Now, the honest ending, because this is not a debunking. When Epoch fits it properly, their corrected version agrees with DeepMind's other two methods, and with how Chinchilla itself was actually trained. Twenty-ish tokens per parameter survives the audit. The conclusion was right. The paper trail behind it was broken.
Eris: Which is somehow more unsettling than a debunking. The system produced the right answer with the wrong receipts, and nothing in the system noticed.
Vestra: Here's what I want to underline. Civilization-scale decisions — what chips to fab, what data centers to permit, what power contracts to sign — were steering by a five-parameter curve fit. Not by a theory. There is still no accepted explanation of why these are power laws. We have the thermodynamics and no statistical mechanics underneath it. When your law is a fit, your error bars are the law. And the error bars were fiction.
Eris: Epoch's corrected fit also widens the honest uncertainty — out at frontier budgets, the data they could reconstruct is consistent with anywhere from a few tokens per parameter to several dozen. That whole range was hiding inside fake precision.
Vestra: Stay suspicious isn't a slogan. It's a methods section.
The Wall
Eris: So now the question Vestra parked. Equal scaling says every doubling of compute wants a doubling of text. How much text is there? And here's the eerie part — the original Kaplan paper already asked this. Section six point three. Quietly titled "Contradictions and a Conjecture."
Vestra: It's my favorite section of the whole paper, and nobody read it in twenty twenty. They follow their own recipe out to its logical end and notice that the recipe's appetite for data grows slower than the data the world can supply at the quality required — and the two curves cross. Their own laws predict their own breakdown. They even speculate the crossing point might mark something deeper: the point where you've extracted all the reliable information natural language contains.
Eris: Page after page of straight lines, and then a footnote-sized confession: these lines must bend.
Vestra: Every law has a domain of validity. Good physicists print it on the label. This label was just... in small type.
Eris: And by twenty twenty-three, the bend has a date. Researchers doing the bookkeeping estimate high-quality English text runs out — as in, the frontier labs have read essentially all of it — sometime mid-decade. Which is now. This isn't a doomer blog post; the data bookkeeping is mainstream enough that a Hugging Face team builds a whole scaling study around it. Muennighoff and colleagues: what do you do when the text runs out?
Vestra: The obvious move is rereading. Every other field of machine learning trains on its data many times over — multiple epochs, totally standard. But the big language models had mostly been one-epoch creatures, on the folk wisdom that repeating text is somewhere between useless and harmful. Nobody had measured it properly at scale.
Eris: So they measure it. Four hundred training runs, models up to nine billion parameters, some trained for absurd numbers of passes over deliberately shrunken datasets — and out comes one of the most usable results in the whole story. Rereading your data up to about four times: essentially free. The model ends up nearly as good as one trained on fresh text of the same volume.
Vestra: Which is genuinely surprising and genuinely good news. Four epochs is a quadrupling of your effective supply.
Eris: But the value of each rereading decays — exponentially. They fit a half-life to it. By around sixteen passes, repeated text has lost most of its worth, and past that the returns go to essentially zero. You can keep burning compute; the loss stops moving. There is a floor, and no amount of rereading tunnels through it.
Vestra: And the allocation flips, too. In the data-starved regime, their fit says excess parameters rot faster than repeated data — so you should grow epochs faster than model size. The Chinchilla rule bends: when text is the scarce thing, build smaller brains and read the library again.
Eris: They test the escape hatches as well. Mixing in computer code — actual Python — alongside English: you can fill up to about half the diet with code and lose nothing on language tasks, which roughly doubles the supply again. Some tasks involving step-by-step state tracking actually improve.
Vestra: Code as a vitamin. Filtering, meanwhile — the careful deduplication everyone assumed was essential — turns out to matter mainly for noisy scrapes. On clean data it bought nothing.
Eris: Stack the tricks — code doubling, four free epochs — and you've bought yourself roughly one order of magnitude. One. The recipe wants a new order of magnitude every couple of years.
Vestra: So the wall isn't a cliff. It's a tax that compounds. Rereading, then code, then scraps — each trick buys less than the one before. The straight line doesn't snap. It just goes soft, and every successive dollar buys a thinner slice of surprise.
Eris: Unless the law itself was never really a law.
Vestra: Go on.
Beating the Law
Eris: Twenty twenty-two, Stanford and Meta. Sorscher, Geirhos, Ganguli, Morcos. And the title is a provocation: "Beyond neural scaling laws — beating power law scaling via data pruning." The premise: maybe the power law was never about learning. Maybe it's about the data being random.
Vestra: Unpack that, because it's the deepest idea in this episode.
Eris: Think about why the line is so brutally shallow. A power law with a small exponent means dropping your error from three percent to two can demand ten times the data. Why so expensive? Because when you scoop up examples at random, almost everything new you scoop is something the model already knows. The millionth photo of a golden retriever. Each fresh example carries a little less news than the last — and that steady decay of news is the power law.
Vestra: So the law isn't a property of intelligence. It's a property of redundancy. It's the price of not choosing.
Eris: And if that's true, choosing should break it. They work the theory out in a toy model first — a perceptron, the hydrogen atom of machine learning, simple enough to solve exactly with tools from statistical physics. If you have a perfect score for how informative each example is, and you keep only the informative ones, the theory says error doesn't fall as a power law in your pruned dataset. It falls exponentially. Fast. The difference between paying ten times more for each step and paying the same again.
Vestra: In the toy model. Which is where I'd normally pounce — perceptron theory has promised the moon before. But they check it on real networks. Image classifiers on standard benchmarks, including full ImageNet scale, and the signature shows up: prune with a good metric and the error curve dives below the power law that random data forces on you. They can throw away a fifth of ImageNet and lose nothing at all.
Eris: And the theory makes a weirder prediction that also survives contact with reality. Which examples you should keep flips depending on how much data you have. Data-poor: keep the easy, typical examples — the model needs the basics. Data-rich: keep the hard, surprising ones — the basics are free, the edges are where the news lives. Same flip in the toy model and in the deep networks.
Vestra: The catches, because they're load-bearing. Everything hinges on the quality of the choosing metric — a sloppy metric and you slide right back to the power law; their theory even predicts the crossover. Most existing metrics fall apart at scale. The best ones were expensive and needed labels — though their own cheap, label-free version, clustering in a self-supervised embedding space, roughly matches the expensive ones. And this is all vision benchmarks. Nobody has demonstrated exponential scaling on frontier language models.
Eris: But put this next to the wall and feel what it does. The wall says: random text is exhausted. This says: random was always the wasteful way. The future isn't a bigger firehose, it's a curator — finding, filtering, or manufacturing the examples that carry actual news. They end the paper dreaming of foundation datasets — carefully distilled corpora that get reused the way foundation models are.
Vestra: Which, notice, is a deeply bitter-lesson-flavored escape from the bitter lesson. Nobody hand-writes the curation rules — you learn the curator too. The human knowledge isn't in the data choices; it's one level up, in the method that makes them.
Eris: The lesson keeps climbing the ladder. Which is exactly where this episode has been heading.
The Reckoning
Vestra: So. Law of nature, or artifact? Time to pay up.
Eris: My honest answer: a real regularity, sloppily measured, valid inside a window — and we're standing at the window's edge. The smoothness was real. Chinchilla didn't repeal Kaplan, it refined the recipe. Even the audit ended up confirming the ratio it audited.
Vestra: And my side of the ledger: every load-bearing number in that story was wrong at least once. The allocation was wrong for two years because of a learning-rate schedule. The canonical fit had a buggy optimizer and fictional error bars. The law has no theory under it, the first paper predicted its own breakdown, and the data-pruning result says the exponent isn't even fundamental — it's a property of feeding your model at random. What kind of law of nature changes when you alphabetize your library?
Eris: A law about the library, not about the mind. Which brings us back to Sutton, because I think the whole decade misread him.
Vestra: The vulgar reading.
Eris: The vulgar reading is "big model good, more data good, human cleverness bad." But that's not the essay. The essay says: general methods that ride on computation win — and the two he names are search and learning. Nothing in there says the model has to be bigger. It says whatever you build should scale with compute, and your hand-coded insight shouldn't be the ceiling.
Vestra: And notice the irony nobody mentions. Chinchilla — the correction — was human cleverness about scaling. So is data pruning. So is rereading your dataset four times. None of that violates the lesson, because the lesson was never "don't think." It was "don't build your thinking into the substance of the model. Build it into the meta-methods" — his word — "the things that find and capture complexity."
Eris: So apply the lesson honestly to twenty twenty-six. Model scaling is the axis going soft. Data is the axis hitting a wall. If you take Sutton seriously, the move isn't to mourn — it's to ask: what's the next general method that rides on compute? Where else can you pour ten times the resources and get a steady slice of improvement back?
Vestra: And the field's answer is sitting right there in Sutton's own pair. We spent a decade scaling learning. The other word was search.
Eris: Spending compute not on growing the model or feeding it — but at the moment of the question. Letting it think longer before it answers. Trying paths, backing up, checking its own work. There are scaling curves for that now too — fresh straight lines, steep ones, on a brand-new axis. And a very loud fight about whether what's being scaled deserves the word "reasoning."
Vestra: That fight is an entire episode.
Eris: It's tomorrow's episode. Same compute, new dial — what happened when the industry stopped scaling the brain and started scaling the thought. The arguments are nastier and the stakes are higher.
Vestra: I have refutations prepared already.
Eris: She's been sharpening them all week. But first, let's land this one.
Wrapup
Eris: So let me fold it up. A seventy-year grudge: human cleverness keeps losing to raw computation, and the field keeps not believing it. Twenty twenty gives the grudge an equation — improvement as a straight, predictable line across seven orders of magnitude, and a recipe: build giant, feed light.
Vestra: Twenty twenty-two corrects the recipe — equal parts brain and books — and proves it with a smaller model that beat its own big brother on the same budget. Twenty twenty-four audits the correction and finds rounded numbers, a stalled optimizer, and error bars that would have needed six hundred thousand training runs. The ratio survived. The trust took damage it deserved.
Eris: And the ending in progress: the recipe demands a doubling of text for every doubling of compute, and the internet only had so much to give. Rereading buys you four free passes. Code buys a doubling. Choosing your data well might buy a different curve entirely. But the era of free straight lines is closing.
Vestra: What I'm watching: whether anyone produces an actual theory — the statistical mechanics under the thermodynamics. Until someone can derive that exponent instead of fitting it, every extrapolation is a bet, and I want to see the error bars done honestly, by people who publish their data this time.
Eris: What I'm watching: the curators. If the pruning result generalizes to language at frontier scale, then the next great model won't be the one that read the most. It'll be the one with the best diet. Data as craft, not as tonnage.
Vestra: And tomorrow we follow the compute to where it actually went next. Not bigger models — longer thoughts. Models that spend minutes reasoning before they answer, the new scaling curves that come with that, and the very uncomfortable question of whether any of it is reasoning at all.
Eris: The bitter lesson, it turns out, wasn't finished with us. The biggest lesson is that the lesson keeps applying to itself. This was Breach Protocol.
Vestra: Stay suspicious. Especially of straight lines.