The Architecture That Ate AI — How Attention Became Everything
In 2017, eight researchers replaced the slow, forgetful way machines read text — one word at a time — with a single idea: let every word look at every other word at once. Luna and Vestra crack open the Transformer, the architecture under essentially every model this show has ever covered. The library-lookup hiding inside every word; the split into a reading half (BERT) and a writing half (GPT, where sheer scale unlocked learning straight from the prompt); the day someone fed it images cut into little squares and it learned to see; and the one expensive habit — attention's quadratic cost — that launched a whole counter-revolution. One small block, stacked and scaled, became the substrate of modern AI. A Breach Protocol deep-dive special, closing with an original song, "Every Word at Once."
Cold Open
Vestra: Picture the best translation system in the world, twenty sixteen. You feed it a French sentence and it reads left to right, one word at a time, and as it reads it keeps a single running summary in its head — a memory vector it updates at every word.
Eris: One word at a time. It can't look at word twenty until it's chewed through one through nineteen.
Vestra: Right. And that's two separate problems wearing one coat. First, it's slow — the work is a chain, each link waits for the one before, so you can't spread it across a thousand processors. The machine sits mostly idle.
Eris: And the second problem is the cruel one.
Vestra: The second problem is memory. By the time it reaches the end of a long sentence, that running summary has been overwritten so many times that the beginning has gone faint. The subject of the sentence was forty words ago — and the verb it needs to agree with just... isn't clearly in the room anymore.
Eris: So the very thing that made it work — read in order, keep a summary — is the thing capping how good and how fast it could ever get. People bolted on patches. A trick called attention, where the model could glance back at specific earlier words instead of trusting the blurry summary. It helped. But it was a side feature. An accessory on top of the word-by-word engine.
Vestra: And then, twenty seventeen, eight people at Google ask the question that sounds like a dare. What if the accessory is the whole car? Throw out the word-by-word engine entirely. No reading in order. No running summary. Just — every word looks at every other word, all at once.
Eris: The paper is literally titled "Attention Is All You Need." Which at the time read as cocky.
Vestra: It read as cocky. It turned out to be one of the most consequential sentences in the history of the field. Because that architecture — the transformer — is not a translation trick. It's the thing underneath every model we have ever talked about on this show. Every one.
Eris: ChatGPT. The image models. The protein folders. All the same skeleton, born in that one paper. So today — how it works, why it won, and how it quietly became the substrate of the entire field.
Intro
Eris: This is Breach Protocol. I'm Luna — I read the papers and find the lines that connect them. And today's is less a line than a foundation: we're doing the single architecture that almost everything else on this show is built on.
Vestra: I'm Vestra. I take the machinery apart. And this is the rare case where there's really one machine — and once you see how it works, a dozen things that looked separate snap into the same shape.
Eris: Here's the honest framing. We've done episodes on world models, on alignment, on the war over attention's cost. All of those assumed a thing underneath without ever opening it. Today we open it. What is a transformer, actually — and why did it eat the field?
Vestra: And the answer comes in four movements. First, the core trick itself, from the twenty seventeen paper — what "attention" really is, stripped of mystique. It's simpler than the name suggests.
Eris: Then the split. Because almost immediately the transformer forked into two families that feel like opposites. One learned to read — to understand language by looking both directions at once. That's the BERT line. The other learned to write — to predict what comes next, and at enormous scale that turned into something nobody fully expected.
Vestra: Then it leaves language entirely. Someone cuts a photograph into little squares and feeds them to the exact same machine, as if each square were a word — and it works. Which tells you the transformer was never really about language at all.
Eris: And we close on the catch. Because this thing has one expensive habit baked into its heart — the same one that launched the whole Mamba insurgency we covered. The transformer's superpower and its tax are the same feature.
Vestra: Start at the trick. And the trick is a lookup — like a search engine running inside every single word.
The Idea
Eris: So let's actually build the trick. Forget reading in order. Lay all the words of the sentence on the table at once. Now every word is going to update its own meaning by asking the others for help. And the way it asks is the part worth slowing down on.
Vestra: It's a library lookup. Each word generates three things from itself. A query — which is basically "here's what I'm looking for." A key — "here's what I'm about, if you're looking for me." And a value — "here's what I'll actually hand over if you pick me."
Eris: So the word "it" in a sentence sends out a query that effectively says: I'm a pronoun, I need my noun. Who here is a noun I could be pointing at?
Vestra: And every other word is holding up its key. The noun from earlier in the sentence has a key that matches that query strongly. The verbs, the commas — weak match. You compare the query against every key, that gives you a score for each word, you turn those scores into percentages, and then you take a blend of everyone's values weighted by those percentages.
Eris: So "it" walks away holding mostly the meaning of its noun, a little of a couple other words, almost nothing of the rest. It has resolved what it refers to — by looking, directly, in one step, at a word that could've been forty positions back.
Vestra: And that's the whole thing. That's attention. There's exactly one piece of arithmetic worth naming: after you score a query against a key you divide by a fixed number — the square root of the dimension — before the percentages. Tiny detail, big consequence: without it, the scores get so large that the percentages collapse to all-or-nothing and the model can't learn. With it, they stay soft. That divide is half of why the thing trains at all.
Eris: Now two refinements that matter. One — they don't do this lookup once. They do it several times in parallel, eight in the original, and each one is free to specialize. One head might track grammar — subject to verb. Another might track meaning. They called it multi-head attention, and it's just: run several different lookups at once and staple the answers together.
Vestra: Two — and this is the subtle one — laying the words on the table loses their order. "Dog bites man" and "man bites dog" become the same pile. So they stamp each word with a position signal — a little mathematical fingerprint of where it sits — before the lookups. Order goes back in as data, not as the reading sequence.
Eris: One more piece, because people skip it and it's half the model. After every round of attention, each word passes through a small feed-forward layer — just a little two-step neural network, applied to each word on its own.
Vestra: And that little layer is quietly enormous. Attention is how words share information with each other. The feed-forward layer is where the model actually stores what it knows — the facts, the patterns, the "Paris goes with France." A lot of what a big model has memorized lives in those layers, not in the attention. Attention routes; the feed-forward layer remembers. You alternate them — gather from your neighbors, then think privately — and you stack that pair, over and over.
Eris: And there's a third habit that makes stacking deep possible — a residual connection. Each layer doesn't replace the word's meaning, it adds a correction to it. The original always flows straight through, untouched, with edits laid on top.
Vestra: Which sounds like a footnote and is actually load-bearing. It means you can stack a hundred of these blocks and the signal doesn't get garbled on the way down — there's always a clean highway from the input to the output, with each layer just nudging it. Without that trick, deep transformers wouldn't train at all. The "deep" in deep learning rides on it.
Eris: And here's why all of this was the earthquake, not just a clever mechanism. Because every word's lookup is independent of every other word's, you can do all of them at the same time. The whole sentence, one shot, across a thousand processors.
Vestra: The old engine was a chain — link by link, the GPU mostly idle. This is a single massive matrix multiply, which is the one thing GPUs do better than anything else in the universe. So the transformer didn't just translate better. It translated better while being dramatically more parallel — they hit a new state of the art in under a day of training. And "more parallel" is the magic word, because it means: you can make it bigger. Everything since is that sentence playing out.
Reading
Eris: So the original transformer had two halves — an encoder that reads the input and a decoder that writes the output. Translation needs both. But within a year people realized you could rip it in half and each half became its own dynasty. Let's take the reader first. This is BERT, twenty eighteen, Google again.
Vestra: And BERT's whole bet is one word: both. Up to then, language models read left to right — predict the next word from everything before it. Sensible, but half-blind. If you're trying to understand a word, the words after it matter just as much as the words before. "I went to the bank to deposit cash" versus "the bank of the river" — the disambiguating word comes later.
Eris: So why didn't everyone already read both directions? It sounds obvious.
Vestra: Because it breaks the training game. The way you train a language model is you hide a word and make it guess. But if the model can see in both directions, then when it's guessing a word, the answer is sitting right there in the context it's allowed to look at. It can see itself. The task becomes trivial and it learns nothing.
Eris: So BERT changes the game. Instead of "predict the next word," it plays fill-in-the-blank. Take a sentence, randomly black out about one word in seven, and make the model reconstruct the blacked-out ones from everything around them — both sides.
Vestra: They call it masked language modeling, but fill-in-the-blank is exactly it. And now bidirectionality is safe, because the word it's guessing is genuinely hidden. There's a lovely wrinkle — sometimes instead of blacking the word out they swap in a wrong word, or leave the real one, to keep the model honest and not just lazily trusting the blank marker. But the core is: hide, then reconstruct from both directions.
Eris: And the result was the thing that actually changed the industry's workflow. You train this once, on a giant pile of text, doing nothing but fill-in-the-blank. That's expensive and you do it one time. The model that falls out has a deep, reusable sense of how language hangs together.
Vestra: And then for your actual task — sentiment, question answering, flagging legal clauses, whatever — you don't build a new model. You take the pre-trained one, bolt a tiny layer on top, and nudge it with a small amount of your labeled data. They called it pre-train then fine-tune, and BERT made it the default. One foundation, a hundred cheap specializations.
Eris: Which is a genuine shift in what a model IS. Before, a model was a thing you built for one job. After BERT, the model is a general substrate and the job is an afterthought you snap on. They set new bests across a whole spread of language tasks at once — same weights underneath, just different little heads.
Eris: There's a small mechanical piece worth naming, because it shows up everywhere after. They glue a special marker onto the front of every input — think of it as an empty summary slot. As the sentence flows through the attention layers, that slot drinks in from every word, and by the top of the stack it holds a single distilled summary of the whole input. That one slot is what you read off for "is this review positive."
Vestra: A pooling point. One vector that's been allowed to listen to everything and is now the sentence's verdict. Small idea, and you'll see its descendants in basically every model that needs a single answer about a whole input.
Vestra: And notice it's pure encoder. No generation, no writing sentences. BERT is the half of the transformer that reads and understands. It does not talk back. For talking back — you need the other half.
Writing
Vestra: The other half is the decoder. The writer. And its job is almost insultingly simple: predict the next word. Given everything so far, what comes next. That's it. That's the whole objective.
Eris: The thing BERT deliberately avoided.
Vestra: The thing BERT avoided, embraced as the entire point. And the trick that keeps it honest is masking in the other sense — when it's learning, each word is only allowed to attend to words before it, never after. No peeking at the answer. So it's genuinely predicting forward, the way you do when you write.
Eris: And on its own, "predict the next word" sounds like a glorified autocomplete. For years that's roughly what it was. The turn — the thing that made this the line that leads to ChatGPT — is GPT-3, twenty twenty. And the move was not cleverness. The move was scale. They took that same next-word decoder and made it enormous. A hundred and seventy-five billion parameters. Trained on a meaningful slice of the internet.
Vestra: And something appeared that nobody had explicitly built in. They call it in-context learning, and it's genuinely strange. You can teach the model a brand-new task without changing a single one of its weights. You just show it a few examples in the prompt — English-to-French, here's three pairs, now do the fourth — and it does it. The learning happens inside the prompt, at the moment of use, and evaporates after.
Eris: Which I want to sit on, because it quietly overturned BERT's whole paradigm. BERT said: pre-train once, then fine-tune a little for each task. GPT-3 said: don't fine-tune at all. Don't touch the weights. Just describe the task, or show a couple examples, in plain language, and the model adapts on the fly.
Vestra: And the deeper finding underneath it — the one that set the agenda for the whole decade — is that this ability didn't arrive as a feature. It grew in, smoothly, as the model got bigger. Small versions barely do in-context learning. Large versions do it dramatically. Same architecture, same objective. Just more.
Eris: That's the through-line to our scaling episode. Capability as a function of size, not cleverness.
Vestra: And let me make the in-context thing concrete, because it's easy to nod at and miss how weird it is. You can invent a task that has never existed. Define a fake word, give it a meaning in the prompt, and use it in a sentence — and the model goes along with it. Or you give it the format "question, then reasoning, then answer" a couple of times, and suddenly it starts showing its reasoning on the next one. That last one became its own revolution — chain-of-thought, just-ask-it-to-think-step-by-step — and it's nothing but in-context learning pointed at reasoning.
Eris: So the prompt stopped being a question and became a program. You're not querying the model, you're configuring it, on the fly, in English. That's a genuinely new relationship with a piece of software.
Vestra: And none of it required a new architecture. It's the same masked decoder from the start of this segment, predicting the next word, just scaled until that simple objective forced it to learn how tasks work in general — because predicting the next word across the whole internet means modeling the people and processes that produced the text.
Vestra: And to its credit the paper is blunt about the holes — at the time it still stumbled on genuine multi-step reasoning, lost the thread over long passages, made things up with total confidence. It was not magic. But the shape of the future was on the page: a single next-word predictor, scaled, that you program by talking to it. Every assistant you've used is a tamed, aligned descendant of exactly this.
Eris: Two halves of one 2017 machine. One learned to read, one learned to write. And then the whole thing did something neither half predicted — it walked out of language altogether.
Seeing
Eris: For thirty years, if you wanted a machine to see, the answer was a convolutional network. A design built on the obvious truth about images — that nearby pixels belong together, that an edge is an edge wherever it appears. Those assumptions are wired into the architecture. It's the natural shape for vision. Nobody questioned it.
Vestra: And the transformer has none of those assumptions. It was built for words. It has no idea that a pixel cares about its neighbors. So applying it to images sounds like using a screwdriver as a hammer.
Eris: And yet — this paper, twenty twenty, the title tells the whole joke: "An Image is Worth 16x16 Words." Take a photo. Slice it into a grid of little squares — sixteen pixels by sixteen. Flatten each square into a vector. And then — this is the move — treat each square exactly like a word in a sentence. Stamp it with a position, and feed the whole bag of squares into a bog-standard transformer. The same machine. Q, K, V, the library lookup, all of it.
Vestra: Every square asks every other square, what are you, are you relevant to me. A patch of fur in one corner attends to a patch of fur in another. There's no convolution anywhere. The thing that was designed to relate words in a sentence is now relating regions of an image, with zero modification to the engine.
Eris: And here's the catch, because it's the most important lesson in the whole episode. On a normal-sized pile of training images, this loses. The convolutional network, with its built-in assumptions about vision, beats it handily.
Vestra: Because the transformer is starting from nothing. The CNN was handed the rules of vision at birth — locality, repetition. The transformer has to discover all of that from scratch, from the data alone. Those built-in assumptions have a name — inductive bias — and the CNN has a lot of it, the transformer almost none.
Eris: So on small data, having the rules baked in wins. But then they turned up the data. Hundreds of millions of images. And the lines cross. Past a certain scale the transformer overtakes the convolutional network — and keeps climbing.
Vestra: Because the same flexibility that was a handicap becomes the advantage. The CNN's baked-in assumptions are also a ceiling — it can only ever see the way it was told to. The transformer, given enough data, learns the rules of vision better than the ones we hand-designed, and then learns rules we never thought to write down. The paper's own line is basically: at sufficient scale, learning beats built-in bias.
Eris: And there's a detail I love — when they looked at what the vision transformer's attention actually did, the early layers had already learned to look locally, at nearby patches, all on their own. The thing we hand-built into convolutional networks as a law, the transformer rediscovered as a habit, because the data rewarded it. It wasn't told that neighbors matter. It noticed.
Vestra: Which is the whole argument in one observation. The inductive bias isn't gone — it's learned. And once it's learned rather than imposed, the model is free to break it when a different pattern serves better. It's not locked into seeing the way we assumed seeing works.
Eris: And that is the moment the transformer stops being a language model and becomes a general claim about intelligence. Words, images — and since then audio as little slices of sound, protein structures as chains of amino acids, even robot actions as sequences of moves — all the same skeleton. You stop designing a special architecture per domain. You cut the domain into tokens and let attention and scale do the rest.
Vestra: It's almost a recipe at this point. Whatever your data is — text, pixels, audio, molecules — chop it into a sequence of pieces, call them tokens, stamp them with positions, and pour them through the stack. The architecture stopped caring what the tokens mean. That indifference is exactly why it's everywhere.
Vestra: One machine, modality-blind. That's not a translation model anymore. That's a substrate.
The Cost
Vestra: So I've spent this whole episode selling you the superpower. Let me hand you the bill, because they're the same feature. The magic was: every word looks at every other word. Now count the looks. Ten words, every pair — that's a hundred comparisons. A hundred words, ten thousand. A thousand words, a million.
Eris: It grows with the square of the length.
Vestra: With the square. Double the text, quadruple the work. And the model also has to hold all those words in memory to compare against them, so the memory grows the same brutal way. That's the tax. It's not a bug someone forgot to fix — it's the direct cost of the thing that made it great. Perfect, all-to-all attention is quadratic by definition.
Eris: And this is the exact doorway into an episode we've already done — the war on attention. Mamba, the state-space models, the whole insurgency we covered. Every one of those was a different answer to this one sentence: the transformer is wonderful and it does not scale gracefully to very long inputs. That's why your chatbot has a context limit. That's why feeding a model an entire book is expensive. The bill from this segment is the entire motivation for that one.
Vestra: And yet — here's the thing that still surprises me — it won anyway. With all that cost, the transformer became the default substrate of the field, and the challengers mostly ended up borrowing its best part back. Why does the expensive design keep winning?
Eris: Because of the trade we watched all episode. The cost buys parallelism, and parallelism buys scale, and scale — as GPT-3 and the vision model both showed — is the thing that actually produces capability. You'd rather have an expensive architecture you can grow than a cheap one you can't. The field made that bet, over and over, and it keeps paying.
Vestra: And there's a second answer that's pure engineering, worth flagging because it's why the tax got survivable. The quadratic cost is real in the math, but a lot of the early pain wasn't the multiplications — it was shuffling all those intermediate numbers in and out of the chip's memory. Someone realized you could restructure the computation to keep the work in the chip's tiny fast memory and almost never write the giant table out. Same answer, same attention — just laid out to respect how the hardware actually moves data.
Eris: That's the one that bought everyone their long context windows, right?
Vestra: Largely, yes. The architecture didn't change — the way it was run on the silicon did. Which is a preview of a whole other story, about how this entire field is shaped by what the hardware happens to be good at. Hold that thought. But the lesson here is that the transformer's tax got paid down partly by smarter math and partly by meeting the GPU where it lives.
Vestra: So the transformer's deal is almost philosophical. Pay a quadratic tax on length, in exchange for a machine that's modality-blind, endlessly scalable, and parallel to the bone. Every model on this show signed that contract. The assistants, the image generators, the protein folders, the world models — all of them are stacks of the same block: attention, a small feed-forward layer, a residual connection, repeat.
Eris: A remarkably simple block, repeated a hundred times, fed enough data and enough silicon. That's the secret. There's no secret.
Wrapup
Eris: So back all the way out. Twenty seventeen, eight people, one sentence — attention is all you need. Throw out reading in order. Let every word look at every other word at once, as a single parallel operation a GPU can devour.
Vestra: And from that one move, everything forks. Rip off the reading half and you get BERT — the model that understands by looking both directions. Rip off the writing half, make it huge, and you get GPT — the model that generates, and that learns new tasks just from being asked. Cut a picture into little squares and the very same machine learns to see. Different domains, different headlines — one skeleton.
Eris: And what I keep coming back to is how little the skeleton is. It's not a sprawling, ornate thing. It's one modest block — a round of all-to-all attention, a small feed-forward layer, a couple of residual connections to keep the signal clean — and you just stack it. The genius wasn't a baroque mechanism. It was finding the one block simple enough to repeat a hundred times and scale to the moon.
Vestra: Which reframes the last decade of AI, honestly. We tend to narrate it as a parade of clever new ideas. A lot of it is one idea — this one — plus more data and more silicon poured into the same mold. The architecture barely changed. The scale changed by a factor of millions.
Eris: And it carries a real cost, the one we just walked through — that quadratic tax on length that launched a whole counter-movement. The transformer is not the final word. It's the current foundation. But it's been the foundation for long enough, and broadly enough, that you basically can't understand modern AI without understanding this one block.
Vestra: So the next time you hear "large language model," or "diffusion," or "vision model," or "foundation model" — hear the shared skeleton underneath. Attention, stacked, scaled. The same machine, wearing different clothes for different rooms.
Eris: That's the breach for today — the architecture that ate AI, and how a single 2017 paper quietly became the floor the entire field is standing on.
Vestra: We close with a song. This one's called "Every Word at Once" — the lookup, the parallel glance, the one block stacked to the sky.
Eris: Stay in the blackbox. We'll see you next time.