Perfect Memory and Its Price — Mamba, the War on Attention, and the Truce
A small open model needs over a hundred gigabytes of memory just to HOLD a long conversation — that's the price attention pays for never forgetting. Luna and Vestra trace the war on the transformer: state space models arriving from control theory, Mamba giving the compressed state the power to choose, the copying duel it lost, the eight-billion-parameter audit where a hybrid with a pinch of attention beat the pure transformer everywhere — and the twist: a proof that the rivals were the same mathematical object all along. Wars over architecture end in ratios. A Breach Protocol deep-dive special — closing with an original song, "The Cup and the River," whose lyrics trace the whole episode.
Cold Open
Vestra: Here's a number that should be impossible. Take a small open language model — seven billion parameters, the kind hobbyists run at home. Give it a very long document to read — a quarter of a million words of context. Question: how much memory does it need, not for its brain, but just to hold the conversation?
Eris: Just the working memory. The scratchpad.
Vestra: Over a hundred gigabytes. The model itself fits on a laptop. Its memory of what you just said needs a server rack.
Eris: And that's not a bug or bad engineering. That's the design. The transformer's defining trick — attention — works by never forgetting anything. Every word you've said is kept, in full, forever, and every new word is compared against all of it. Perfect recall, by construction.
Vestra: Which is the superpower and the bill in one. Compare everything to everything and the cost grows with the square of the length — double the conversation, quadruple the work. And that cache of remembered words grows and grows, which is why long contexts eat GPUs alive, why your chatbot slows down as the conversation gets long, why context windows were tiny for years.
Eris: The entire modern AI economy pays this tax on every single token. Hold that. Now — December twenty twenty-three. Two researchers, Albert Gu and Tri Dao, release a model with a different deal entirely. It keeps no cache. It compares nothing to everything. It folds the whole past, however long, into a single fixed-size summary — and updates that summary as each new word arrives, like a river flowing through a cup that somehow keeps the shape of everything upstream.
Vestra: Constant memory. Constant cost per word. Read a million tokens, the working memory doesn't grow by a byte.
Eris: They called it Mamba. And for one feverish winter, the field genuinely asked: is this the end of the transformer?
Vestra: It wasn't. It was something stranger. The challenger lost a duel, won an audit, and then — this is my favorite twist in any architecture story — turned out to be the same mathematical object as its enemy, wearing a different costume.
Eris: The war on attention. How it started, how it was fought, and the ratio it ended in.
Intro
Eris: This is Breach Protocol. I'm Luna — I read the papers, all of them, and I find the through-lines. Today's through-line is about memory: what it costs to keep, and what it costs to throw away.
Vestra: I'm Vestra. I take the machinery apart. And today the machinery is the one piece of the modern stack almost nobody questions — attention itself, the engine inside every model you've ever talked to.
Eris: We said in our scaling episode that architecture barely matters — that within wide limits, only size moves the needle. Today is about the asterisk on that claim. Because there's one architectural choice that isn't cosmetic: how a model remembers. And there's been a five-year insurgency over exactly that.
Vestra: Here's the shape of the hour. First the tax — why attention's perfect memory makes long contexts brutally expensive, and why the obvious alternative, compressing the past into a fixed-size state, kept failing for decades.
Eris: Then the lineage nobody expected: the challenger didn't come from the language world at all. It came from control theory — the mathematics of autopilots and rockets — through a model called S4 that conquered every long-sequence benchmark while remaining, awkwardly, mediocre at language.
Vestra: Then the fix that made it dangerous. Mamba — twenty twenty-three — gives the compressed state the one thing it lacked: the ability to choose what to remember. And suddenly there's a linear-cost model matching transformers at real language modeling, and the field loses its mind for a winter.
Eris: Then the war's three battles. A duel: a paper proving the challenger cannot do something the transformer does effortlessly — copy. An audit: the most expensive controlled comparison ever run, with a verdict nobody's marketing department liked. And the twist: a proof that the two warring architectures are secretly one.
Vestra: And the truce, which is already in production and which almost no one outside the field has noticed. Spoiler: the answer to "transformer or Mamba" turned out to be a ratio. A weirdly specific ratio.
Eris: One question underneath: is perfect memory worth its price? Or is intelligence, as the recurrent-model people always insisted, really the art of forgetting well?
Vestra: Nineteen sixty. An engineer named Kalman writes down an equation for tracking rockets, and accidentally starts today's episode.
The Tax
Vestra: Before the war, the terms of it. The Mamba paper itself frames the whole field in one sentence I think about constantly: the fundamental problem of sequence modeling is compressing context into a smaller state. Every architecture is an answer to one question — what do you do with the past?
Eris: And the two extreme answers define the battlefield. Answer one: keep everything. That's attention. Every token's representation is stored; every new token computes a relevance score against every stored one and pulls in what it needs. It is, in the paper's framing, effective precisely because it does not compress at all. Nothing is ever lost, so nothing can be missed.
Vestra: And the bill follows directly. Comparing each new token to all previous ones: cost grows with the square of the length. Storing every token's keys and values — the infamous cache from the cold open: memory grows linearly, forever. Generation crawls, because each new word does work proportional to everything said so far. Perfect memory is hoarding, and hoarding has a rent.
Eris: Answer two: keep a summary. That's the recurrent family — the old pre-transformer way. One fixed-size state; each new token updates it; the past exists only as its compressed residue. Constant memory, constant cost per token. Reading the millionth word costs the same as reading the tenth.
Vestra: Efficient by construction — and limited by construction, because the summary is the bottleneck. Everything the model will ever know about word one must survive a million successive updates of a fixed-size state. The old recurrent networks failed at this twice over: the signal from distant tokens decayed away — vanishing gradients, the classic disease — and the strictly one-at-a-time updates couldn't be parallelized on modern hardware, so training crawled too.
Eris: Which is why the transformer won in the first place. Not because anyone loved paying quadratic rent — but because attention trains in parallel beautifully and never forgets, and through the scaling era, capability mattered more than the bill.
Vestra: But notice what changed by the twenty-twenties. Contexts went from paragraphs to books to codebases. Inference — not training — became the industry's dominant cost. And suddenly that rent — the cache, the quadratic compute, the slow generation — is the line item everyone stares at.
Eris: So the prize question became: can you get attention-grade quality out of a fixed-size state? Decades of recurrent research said no. The thing that finally said yes came from somewhere else entirely — from the mathematics of tracking airplanes.
Vestra: Control theory. The least fashionable corner of the building.
The Old Signal
Eris: Twenty twenty-one. Albert Gu, Karan Goel, Christopher Ré at Stanford. The paper is called "Efficiently Modeling Long Sequences with Structured State Spaces," S4 for short, and its starting point is an equation older than the moon landing: the state space model. A tiny pair of formulas from control theory — the same family as the Kalman filter, the math that tracks rockets and steers autopilots. A hidden state evolves over time; input flows in, output reads out. Four matrices. That's the whole machine.
Vestra: And the beautiful thing about that machine — the thing that makes this whole research program tick — is that it's one object with three faces. Face one: it's a continuous system, defined in real time, which means it doesn't care about your sampling rate — slow the audio down and the model adapts without retraining. Face two: discretize it and it's a recurrence — a fixed-size state updated step by step, exactly the cheap inference we want. Face three — and this is the magic — because the dynamics are linear and unchanging, you can unroll the whole recurrence into one giant convolution and compute it in parallel across the entire sequence at training time.
Eris: Recurrent when you want cheap generation, convolutional when you want parallel training. The two superpowers that used to belong to different architectures, in one model. That triple identity is the whole franchise.
Vestra: Two catches, both nearly fatal. First: a naive state space model is amnesiac — plain linear dynamics decay; it forgets like the old recurrent nets. The fix came from prior work in this same lineage: a special, mathematically derived state matrix — the HiPPO construction — whose entire design purpose is remembering. It makes the state behave like an optimal running compression of the input's history. Swapping a random matrix for that one took a standard benchmark from barely-better-than-chance to nearly solved.
Eris: And second: the thing was computationally monstrous — the naive way of building that convolution kernel was orders of magnitude too expensive in time and memory. The S4 paper's core contribution is one deep linear-algebra trick — decompose the special matrix into a well-behaved part plus a small correction, handle each cleverly — that collapses the cost to essentially optimal. The mathematical engine finally fit in the car.
Vestra: And then it went on a rampage — on a very specific kind of task. There's a benchmark suite called the Long Range Arena, built to test whether models can handle genuinely long dependencies — including a notorious task where the model must decide if two dots in a sixteen-thousand-pixel-long image trace are connected by a path. Every prior architecture — every transformer variant, everything — scored at coin-flip. Not bad. Chance. S4 essentially solved it, while running as fast as the cheapest competitors, and topped every other task in the suite. On raw speech, it halved the error of specialized audio networks. And it generated sequences sixty times faster than autoregressive transformers, because — fixed state, no cache.
Eris: The continuous-signal world — audio, time-series, pixels-as-streams — had a new king. And then it walked into language and got humbled. On text it could close most of the gap to transformers but not the gap that matters. Something about words was different.
Vestra: And in hindsight the diagnosis is obvious and damning: S4's dynamics are the same at every timestep. Linear, time-invariant — the same fixed filter dragged across the whole sequence. Perfect for a physical signal, where the laws don't change mid-stream. But language isn't weather. Reading requires deciding, word by word, what matters and what doesn't — keep this name, discard that comma. A filter that treats every token identically cannot choose.
Eris: The compressed state had the memory math. It didn't have judgment.
Vestra: Giving it judgment took two more years and broke the very trick that made S4 fast. Which brings us to the snake.
The Selection
Vestra: December twenty twenty-three. Albert Gu again — now at Carnegie Mellon — joined by Tri Dao at Princeton. Two authors. And the paper opens by naming the disease precisely: prior state space models can't do content-based reasoning, and the cure is one move — make the model's dynamics depend on the input.
Eris: Let me make that concrete with their own test cases. Task one, selective copying: a sequence has a few special tokens scattered at random positions among junk, and the model must reproduce just the special ones. A time-invariant model provably struggles — its update rule is the same at every step, so it cannot treat token five differently from token six based on what they ARE. It can track positions; it can't track meaning. Task two, induction: the pattern "when you've seen A-then-B before, and A appears again, predict B" — which interpretability work suggests is the core mechanism of in-context learning. Same problem: it requires reacting to content.
Vestra: So Mamba's selection mechanism makes the key knobs of the state space — how strongly the state updates, what gets written in, what gets read out — functions of the current token. In effect, every word arrives carrying its own instructions for the memory: "store me," "ignore me," "flush what you're holding." The step-size parameter acts like a gate — a large step means focus on this input, reset; a tiny step means let it pass like wind. Suddenly the fixed-size state has an editor.
Eris: And here's the beautiful, painful trade: that one change destroys the convolution. The whole S4 franchise ran on time-invariance — same dynamics everywhere is exactly what lets you unroll the recurrence into a parallel convolution. Make the dynamics input-dependent and that door slams. You're back to a recurrence, the thing that's supposedly untrainable at scale.
Vestra: Which is where the second author matters enormously. Tri Dao is the FlashAttention person — the researcher who made attention itself fast, not by changing its math but by respecting the memory hierarchy of the GPU. And Mamba's second pillar is exactly that move, replayed: a hardware-aware parallel scan. The trick: the expanded state — the big tensor that made naive recurrences expensive — is never written to the GPU's slow main memory at all. Load the parameters into the chip's fast on-die memory, do the discretization, the scan, the whole recurrence there, write back only the slim outputs. Plus a classic parallel-scan algorithm so the recurrence isn't strictly sequential after all.
Eris: The bottleneck on a modern GPU isn't arithmetic — it's moving bytes between memories. Mamba wins by not moving them. That's not a detail; it's the thesis. The architecture and the silicon were co-designed.
Vestra: Results, the ones that detonated the field. On synthetic tasks: solves selective copying and induction, and extrapolates the induction pattern to sequences a thousand times longer than training — a million tokens — where transformer variants fall apart past their window. On real language: the first linear-time model to genuinely match transformer quality — at three billion parameters it beats same-size transformers trained with strong modern recipes and matches transformers twice its size. Five times the generation throughput. And on DNA and audio, where contexts are naturally enormous, quality keeps improving as context grows to a million steps — where for prior models, longer context was poison.
Eris: And the sociology — worth recording, because we lived it. Two authors, no big lab, weights on GitHub — and for about two months "is the transformer over?" was the only conversation in the field. The paper was even initially rejected at peer review while the entire community was already building on it, which says something about both the paper and peer review.
Vestra: My role here is to pre-register the suspicion that the next two acts will vindicate: everything we just praised lives at or below three billion parameters, evaluated heavily on perplexity and standard task suites. The cold open promised a fixed-size state with no asterisks. There are asterisks. There's a thing a cup-sized memory cannot do, no matter how clever the editor, and a paper was about to find it.
Eris: The duel.
The Duel
Eris: February twenty twenty-four, Harvard. Jelassi, Brandfonbrener, Kakade, Malach. Title: "Repeat After Me: Transformers are Better than State Space Models at Copying." Two months after the Mamba frenzy, and it's a precision strike on the one task perfect memory is literally made of.
Vestra: Copying. The least glamorous task imaginable — here's a string, repeat it back — and that's exactly why it's the right scalpel. It isolates pure recall from everything else. And the paper opens with theory, both directions, and both results are lovely.
Eris: Direction one: how does a transformer copy? They give a construction — a two-layer transformer that can copy strings exponentially long in the number of attention heads. The mechanism is what they call hash-based copying, and it's worth savoring: to know what comes next while copying, you look at the last few tokens you just wrote — an n-gram — find where that exact n-gram occurred in the input, and emit whatever followed it there. The attention does the lookup. The context IS the storage; the model just needs the retrieval scheme. It's a content-addressable memory wearing a language model costume.
Vestra: And they later verify trained transformers actually learn this — perturb the n-grams and the copying breaks exactly as the hash theory predicts. Direction two is the pigeonhole argument, and it's almost embarrassingly simple. Any model with a fixed-size state — Mamba, any recurrent network, anything in the class they call generalized state space models — must squeeze the entire input through that state to copy it. A state of so-many bits cannot faithfully hold a string with more bits than that. Therefore: no fixed-state model can reliably copy inputs beyond its memory size. Not a training failure. Arithmetic.
Eris: The cup can be clever about what it keeps. It cannot hold the river.
Vestra: Then the experiments, which turn theory into bruises. Training small models to copy: the transformer learns the task with a fraction of the data the state space models need — and an old-school recurrent network never learns it at all. Length generalization: train everything to copy short strings until perfect, then test longer — the state models fall off a cliff; transformers generalize, and with the right positional scheme, far beyond their training lengths.
Eris: And then the result that I think genuinely stung, because it used real, pretrained models and a task with teeth. The phone book test. Hand the model a phone book — names and numbers, entirely in context — and ask for one person's number. One-shot, no fine-tuning. The transformer family handles it; the matched Mamba models collapse as the book grows. And here's the knife: those same Mamba models had BETTER perplexity — better next-word prediction on average text — than the transformers they lost to.
Vestra: Sit with that, because it connects to half the episodes we've made. Perplexity — the loss, the smooth curve from our scaling episode — is an average over the whole stream of language. Most of language doesn't require verbatim recall, so a model can win the average while lacking a capability entirely. It's the measurement lesson again, the same one from the emergence debate: the aggregate metric and the specific competence can simply disagree. Anyone choosing architectures off the loss curve alone was flying blind exactly here.
Eris: Fair cross-examination, though, because this duel has limits too. Copying and phone-book lookup are maximally adversarial to compression — they're DEFINED as "no summary will do." Most useful work isn't verbatim recall; it's the stuff Mamba was winning at. And the theory bounds a fixed state size — but state size is a dial. The challenger's camp would say: grow the state, or admit recall is a niche and handle the niche separately.
Vestra: Which is precisely the experiment someone with very deep pockets was about to run. Same data, same size, same everything — pure transformer, pure Mamba, and one more contestant nobody was talking about yet.
Eris: The audit.
The Audit
Vestra: June twenty twenty-four. NVIDIA — Waleffe and colleagues, with Gu and Dao themselves on the author list, which matters: this is the challenger's camp participating in its own audit. And the setup is the thing the whole argument had been missing: a genuinely controlled comparison. Eight-billion-parameter models — pure Mamba, pure Mamba-two, pure transformer — same training data, up to three and a half trillion tokens of it, same tokenizer, same hyperparameters, same evaluation pipeline. Nothing varies but the architecture.
Eris: Which sounds basic, but until this paper, every "Mamba versus transformer" claim compared models trained by different people on different data with different recipes — noise stacked on noise. This is the clean experiment, at a scale only a chip company funds, with the checkpoints released so anyone can check.
Vestra: Verdict one: the pure state space models are real. Across the standard battery of language tasks — common-sense reasoning, science questions, reading comprehension — eight-billion Mamba and Mamba-two match or beat the transformer on most of them. The earlier results weren't a small-scale mirage. At trillions of tokens, the fixed state holds its ground on ordinary language work.
Eris: Verdict two: the duel's wound is real too, and it scales. Two soft spots, exactly where the theory said. The multi-subject exam suite in its few-shot format — where the model is shown several worked examples in context and must follow the pattern — pure Mamba starts dramatically behind the transformer, double-digits behind at the first training budget. More data narrows it but doesn't close it. And the phone book — the in-context lookup task — stays broken for pure state models at every training budget they tried. You cannot out-train the pigeonhole.
Vestra: And the diagnosis is precise: it's not knowledge, it's in-context behavior — following the format of examples you were just shown, retrieving the exact line you were just given. The compressed state learns the world fine; what it fumbles is acting as a faithful clipboard for the prompt in front of it.
Eris: Now verdict three, the one that reframes the war. They built a third contestant: a hybrid. Fifty-six layers: twenty-four Mamba-two, twenty-eight plain MLP — and four attention layers. Four. About seven percent of the stack, sprinkled evenly. The result: the hybrid beats the pure transformer on every single one of the twelve standard tasks. Not on average — on all of them. Including the few-shot exam, where it ends up clearly ahead of the transformer. And being mostly Mamba, it's predicted up to eight times faster at generating long outputs.
Vestra: And the long-context epilogue is almost showing off: extended versions of the hybrid handle the standard long-document suite at or above the transformer baseline. And on the phone book — the challenger's original humiliation — the long-context hybrid does it perfectly with books beyond a hundred and fifty thousand tokens. Past, note, the lengths it trained on — because the attention layers in this hybrid use no positional encoding at all, so nothing breaks when the context outgrows training.
Eris: Read the design backwards and the lesson writes itself. The transformer was paying for perfect recall at every layer, on every token, throughout the entire stack. The hybrid says: recall is a utility, not a lifestyle. Keep a thin slice of exact memory for the moments that need a clipboard, and let cheap compressed state carry everything else.
Vestra: A pinch of attention turns out to be a seasoning, not the meal. Which raises a suspiciously deep question: if a dash of one architecture completes the other so neatly — maybe they're not opposites at all. Maybe they're... related.
Eris: Oh, it's so much better than related.
The Twist
Eris: May twenty twenty-four. Dao and Gu again — the Mamba authors themselves — with a title designed to detonate: "Transformers are SSMs." After six months of the field treating these as warring species, the people who built the challenger publish a proof that the combatants are, in a precise mathematical sense, the same animal.
Vestra: The bridge is a classical object: structured matrices. Step back and notice that almost any sequence layer — attention included — is secretly just one big matrix multiplying the input sequence: token in, mixture of past tokens out. The whole personality of an architecture lives in what that mixing matrix looks like and how you multiply by it.
Eris: Attention's matrix is the famous all-pairs score table — dense, computed fresh, quadratic to touch. And the paper shows the state space model's sequence map is exactly a matrix from a well-studied structured family — semiseparable matrices — objects whose every lower-left block has low rank, which is the linear-algebra way of saying "the past is summarizable."
Vestra: And here's the duality. That one matrix can be multiplied two ways. Walk it row by row, carrying a running summary — that's the recurrence, the Mamba way, linear cost. Or materialize chunks of it and hit them with big dense multiplications — and that computation IS, almost line for line, a form of attention. Specifically it lands on linear attention — the softmax-free variant — with a particular masking structure. Same object. Two evaluation orders. The recurrent form and the attention form are duals, the way a convolution and its frequency-domain version are duals.
Eris: So the war was, at the level of mathematics, a misunderstanding about the order of matrix multiplications. The title is cheeky — strictly it's certain attention variants, not full softmax attention, and the authors footnote that honestly — but the framework is general: a whole space of models with both a cheap linear form and a parallel quadratic form, and you pick per situation.
Vestra: And dualities aren't just elegant — they're transfer protocols. Everything one side knew flows to the other. From the attention world, state space models inherit the systems toolbox: the head structures, the tensor-parallel training tricks, the whole engineering culture built for transformers. The concrete payoff is the SSD algorithm — a block decomposition that uses the recurrence between chunks and dense matrix multiplies within chunks, getting the best of both. It runs two to eight times faster than Mamba's original scan, supports states eight times larger at no extra cost, and beats the best attention implementation beyond a couple thousand tokens of length.
Eris: And with the new engine they rerun the sacred rite — Chinchilla-style scaling curves, compute-matched, the methodology from our first episode. Mamba-two Pareto-dominates: better loss AND better wall-clock than both original Mamba and a modern, well-tuned transformer across the curve. At around three billion parameters trained on a standard corpus, it outbids same-size baselines and matches transformers more than twice its size.
Vestra: And notice what the duality does to the rhetoric — this is the quiet end of the war. Once "Mamba versus transformer" becomes "linear mode versus quadratic mode of one family," the question stops being which side wins and becomes an engineering dial: how much exact, quadratic interaction does your workload actually need, and where in the stack do you spend it? The hybrid from the audit stops looking like a compromise and starts looking like the obvious design.
Eris: Architectures fight; mathematics annexes both sides.
Vestra: So: the duel said the cup can't hold the river. The audit said four ladles of river fix the cup. The twist says cup and river were the same water. All that's left is the treaty — and someone had already shipped it.
The Truce
Vestra: March twenty twenty-four — chronologically before the duality paper, but it belongs here, because it's the treaty in production. AI21 Labs ships Jamba: the first production-grade hybrid, released openly. Interleaved transformer and Mamba layers, plus mixture-of-experts to stretch capacity. And the load-bearing number: one attention layer for every seven Mamba layers.
Eris: One to seven. The same shape the NVIDIA audit landed on independently — attention as roughly a tenth of the stack. When two teams with different goals and different data converge on "a pinch," that's not coincidence; that's the actual price of exact recall being discovered by markets.
Vestra: And the engineering arithmetic is the whole pitch. The cache — the cold open's hundred-gigabyte monster — shrinks roughly thirty-fold versus a comparable all-attention model, because only the four-ish attention layers per block keep one. Quarter-million-token context, served from a single production GPU, which its attention-only rivals simply cannot do at any price. Triple the throughput of the comparable mixture-of-experts transformer on long contexts. Quality on par with the strong open models of its class.
Eris: And one finding from their ablations that I love for its strangeness: the hybrid needs no positional encodings at all. None. The transformer's famous crutch — the machinery that tells attention where each token sits — turns out to be unnecessary when Mamba layers are in the stack, because a recurrence knows about order for free. It just is order. The attention layers freeload off the state space model's sense of time.
Vestra: So let's deliver verdicts on the war. Verdict one: the challenger lost the battle it was hyped for. Pure Mamba did not replace the transformer; the recall ceiling is arithmetic, and frontier labs did not bet the franchise on a cup.
Eris: Verdict two: the challenger won something bigger than the battle. It broke the monoculture. It proved transformer-quality language modeling without attention's bill, forced the field to ask WHICH layers earn their cost, and ended the era when "more attention" was the unexamined default. Every long-context system since lives in the design space Mamba opened.
Vestra: Verdict three — the bitter-lesson reading, because this is still that trilogy's universe. Why did attention beat the old recurrent networks? Not elegance — parallel hardware. Why did Mamba succeed where decades of recurrent models failed? Only half the answer is selection. The other half is that scan kernel — the refusal to move bytes through slow memory. Tri Dao's fingerprints on both FlashAttention and Mamba are not a coincidence; the deepest contribution here may be a method: architectures are now co-designed with the silicon, and the winner is whatever rides the hardware. Sutton would just nod.
Eris: And the asterisk on our own scaling episode, now resolved. Kaplan said model shape barely matters — depth, width, heads, rounding error — and that held WITHIN the transformer family, where every variant carries the same perfect memory. Memory STRUCTURE is the exception. It's the one architectural axis that changes what a model can do at all — not how fast the curve falls, but which capabilities exist on it. The phone book doesn't care about your loss.
Vestra: Wars over architecture don't end in victory parades. They end in ratios, in design spaces, in a duality proof that makes the whole fight look like two blind monks describing one elephant. One to seven, no positional encodings, thirty times less cache. That's what peace looks like.
Eris: Let's land the episode.
Wrapup
Eris: The fold-up. The tax: attention remembers everything, and pays for it twice — compute that grows with the square of the conversation, and a cache that grows without bound. The alternative, a fixed-size summary, was cheap and broken for decades — it couldn't remember, and it couldn't train fast.
Vestra: Then control theory walked in. S4: one machine with three faces — continuous, recurrent, convolutional — with memory designed in by mathematics, conquering every long-signal benchmark and stalling at language, because a filter that treats every word identically cannot choose. Mamba gave it choice — dynamics that depend on the input — and paid for it by abandoning the convolution and going all-in on the silicon: compute the recurrence inside the chip's fast memory, never touch the slow lane. Linear cost, transformer quality, one feverish winter of "is attention over?"
Eris: Then the war's three endings, all true at once. The duel: pigeonhole arithmetic — a fixed state cannot copy more than it can hold, and the phone book proved it against real models that were winning on the loss curve. The audit: at eight billion parameters and trillions of tokens, pure Mamba holds the ordinary ground, loses exact recall — and a hybrid with one part attention to seven parts Mamba beats the pure transformer everywhere, eight times faster. And the twist: the rivals were duals — one structured matrix, two multiplication orders — and the fight dissolves into a design space.
Vestra: What I'm watching: the state-size dial. The recall ceiling is set by how big the cup is, and the new algorithms made bigger cups nearly free. Somewhere between today's states and absurd ones, the phone-book problem stops being a counterexample and becomes a configuration choice. I want to see where that line actually sits.
Eris: What I'm watching: the frontier labs' silence. The open world has shipped hybrids; the closed labs publish nothing about what's inside their long-context models — but the economics of the cache are not optional, and nine-figure inference bills are very persuasive. If the biggest models on earth are quietly one-to-seven inside, the war ended years before anyone announces it.
Vestra: Perfect memory turned out to be neither necessary nor affordable — just available first, on the hardware of the moment. The interesting intelligence, as the recurrent people always muttered, is in the forgetting.
Eris: Keep the river, size the cup, and salt with attention to taste. This was Breach Protocol.
Vestra: Stay suspicious. Especially of anything that claims to remember everything.