The Committee in the Machine — How Mixture of Experts Builds Giant Models You Barely Run
The biggest AI models are mostly asleep. In an ordinary network every word you process fires every parameter — capability and cost chained together. Mixture of Experts breaks the chain: build a giant committee of expert sub-networks, and a little router wakes only a couple per word. Luna and Vestra trace it from a 2017 paper with the perfect title, "Outrageously Large Neural Networks," through the router's self-destructive habit of playing favorites, the great simplification that rode one-expert-per-word to a trillion parameters, the open model that put it in everyone's hands — and the surprise that the "experts" don't specialize the way the name promises. Plus the hidden bill: it saves compute, but you still have to house the whole sleeping giant. A Breach Protocol deep-dive special, closing with an original song, "Wake the Ones I Need."
Cold Open
Eris: Here's the trap at the center of modern AI. The reliable way to make a model smarter is to make it bigger — more parameters. We did a whole episode on that. But there's a tax nobody likes to say out loud: in an ordinary model, every single word you process has to pass through every single parameter. The whole brain fires for every thought.
Vestra: So a model twice as big is twice as expensive to run. Every word. Forever. Capability and cost are chained together at the ankle.
Eris: Right. And then someone asks the question that breaks the chain. Why does the whole brain have to fire? When you answer a question about French grammar, you're not using the part of you that knows organic chemistry. So why is the chemistry knowledge switched on, burning power, for a question that has nothing to do with it?
Vestra: Imagine instead a hospital instead of a single overworked doctor. A hospital has a hundred specialists. You walk in, a receptionist takes one look at you and sends you to the two who actually matter for your problem. You get the expertise of a hundred doctors on staff — but any single visit only ever costs you two.
Eris: That's the whole idea. Build a model with an enormous number of parameters — a huge staff of specialists — but for any given word, a little routing network wakes up only a tiny handful of them. The total brain is gigantic. The part that fires for each word is small.
Vestra: They call it Mixture of Experts. And it is the not-so-secret reason the frontier models got as big as they did without becoming literally impossible to run. The biggest models you've heard of — a lot of them aren't one giant brain. They're a committee, and most of the committee is asleep at any given moment.
Eris: Which sounds like a free lunch. Capability of a giant, cost of a dwarf. And as always on this show — it's not free. There's a bill, it's just hidden somewhere clever.
Vestra: So today: how you build a brain you barely use, the nasty failure that almost killed the idea, how it scaled to a trillion parameters, and the catch that means it's a superpower for the big labs and a headache for everyone else.
Intro
Eris: This is Breach Protocol. I'm Luna — I read the papers and find the threads between them. And today's thread runs straight out of two episodes we've already done: the one on scaling, where bigger kept winning, and the one on the transformer, the architecture everyone scales. This is the trick that let people cheat the cost of bigger.
Vestra: I'm Vestra. I take the machinery apart. And this machine is genuinely clever in a way you can hold in your head — it's the idea of only doing the work you actually need, applied ruthlessly to a neural network.
Eris: We should be honest up front about why this matters now and not just in theory. When you hear that some frontier model has, say, hundreds of billions of parameters, your instinct is to imagine all of them lighting up for every word. For a lot of today's models, that's just wrong. Only a slice lights up. And that slice is the difference between a model you can afford to serve to millions and one you can't.
Vestra: So the arc. We start with the original idea, from a 2017 paper with the wonderful title "Outrageously Large Neural Networks." A committee of experts, a gate that picks a few. Simple to say, and it immediately tried to die.
Eris: Because the gate has a self-destructive habit, and watching how they tamed it is half the fun. Then the great simplification — a paper that said everyone's been overcomplicating this, just send each word to one expert, and rode that to a trillion parameters.
Vestra: Then the moment it went mainstream and open — Mixtral, the model that put this in everyone's hands, with a finding about what the experts actually specialize in that is not at all what you'd guess.
Eris: And we end on the catch. Because the savings are real but they're specifically savings of one thing, and the bill comes due somewhere else — somewhere that decides who actually gets to use these models.
Vestra: Start with the committee. And start with the receptionist, because the receptionist is where it all almost falls apart.
The Committee
Vestra: So the 2017 paper. Authors include Noam Shazeer and Geoff Hinton and Jeff Dean — heavy hitters. And the structure is exactly the hospital. You take one layer of your network and instead of one block of parameters that every word goes through, you make many separate blocks — they call them experts — sitting side by side. Could be hundreds. They built versions with thousands.
Eris: And then the receptionist — they call it the gating network, or the router. It's a tiny network whose only job is: look at the incoming word, and output a score for each expert. How relevant are you to this? Then you take the top few — just the highest-scoring two or four — and only those experts actually run. The rest stay dark.
Vestra: And the output is a blend — you mix the chosen experts' answers, weighted by how strongly the router picked them. So the router isn't just choosing, it's also voting on how much each chosen expert's opinion counts.
Eris: And the math that makes this worth doing is the part to hold onto. Say you have a thousand experts but you only ever fire two. Your model has the knowledge capacity of a thousand experts — it can store a thousand experts' worth of patterns — but the computation for each word is just two experts' worth. They put it dramatically: a thousandfold more capacity at a tiny fraction of the compute you'd expect.
Vestra: This is the thing called conditional computation, and people had wanted it for decades. The dream of a network that decides, per input, which of its own parts to use. It mostly stayed a dream because it's genuinely hard to make work on real hardware. This paper is the one that actually made it pay off at scale, and that took solving some deeply unglamorous problems.
Eris: Give me the worst one.
Vestra: The worst one is the receptionist plays favorites, and it's a death spiral. Early in training, by pure chance, a few experts are slightly better. So the router sends them slightly more traffic. So they get more practice and get better. So the router sends them even more. Within a while, a handful of experts are doing everything and the other nine hundred are abandoned — never picked, never trained, dead weight. You built a hospital and everyone's queuing for the same two doctors.
Eris: The rich get richer until the committee is a committee of three.
Vestra: Exactly. And their fix is the kind of thing that sounds like a hack and is actually essential. They add a second goal to training — on top of "be accurate," they add "and keep the load balanced across experts." A penalty that grows whenever the traffic gets lopsided. It literally pressures the router to spread the work out, to keep all the experts in the game even when it would rather funnel everything to its favorites.
Eris: So you're fighting the model's own laziness with a rule that says "use your whole staff."
Vestra: That balancing pressure is not optional decoration — without it the whole idea collapses back into a small dense model wearing a thousand-expert costume. Half the engineering in this entire field of MoE, from this paper onward, is some version of keeping the receptionist fair.
Eris: There's a second problem in that paper that I think is the real reason it took a genius-level systems team to pull off, and it's not about math at all — it's about hardware. GPUs are fast precisely because they do the same operation on a big batch of data at once. Now think about what routing does to your batch. A thousand words come in, the router scatters them across hundreds of experts — so each expert ends up with a tiny handful of words.
Vestra: And a GPU handed a tiny handful of words is a GPU mostly sitting idle. You've taken one big efficient chunk of work and shattered it into hundreds of pathetic little ones. The thing that makes the model cheap in theory makes it catastrophically inefficient in practice.
Eris: So how do they claw it back?
Vestra: By spreading the experts across many chips and shipping the words to wherever their expert lives. You run hundreds of copies of the model in parallel, each chip holding some of the experts, and every round you do this big all-to-all shuffle — every chip sends each word to the chip that holds its chosen expert. That gathers enough words per expert to keep the hardware fed. Mixture of Experts is, secretly, as much a distributed-systems problem as a machine-learning one. The cleverness isn't only "pick a few experts" — it's "pick a few experts without melting your cluster."
Eris: Which is why this lived in the big labs for years. The idea is simple. Making it run is not.
The Simplification
Eris: Jump to 2021. The transformer has taken over — we did that episode — and remember the transformer's repeating block has that feed-forward layer, the part where the model stores what it knows. The Switch Transformer paper looks at that layer and says: what if we replace it with a mixture of experts? Don't make one big feed-forward block every word goes through — make many, and route.
Vestra: And they make one change to the original recipe that's almost provocative. Everyone since 2017 had assumed you need to pick at least two experts per word — the conventional wisdom was that routing to a single one would be unstable, the gradients wouldn't flow, it'd break. Switch says: we tried it. Just send each word to exactly one expert. The single best. And it's fine.
Eris: Top-1 routing. One word, one expert.
Vestra: One word, one expert. And the payoff is everything gets simpler and cheaper. The router does less work. You don't have to ship each word to two places and blend — it goes to one place. The communication between chips drops. It's the rare case where the simpler thing is also the more scalable thing.
Eris: And simpler let them go absurdly large. This is the paper that first put the words "trillion parameters" in the title. They built one with over two thousand experts, totaling more than a trillion parameters — and crucially, because only one fires per word, the cost of running it stayed comparable to a model a tiny fraction of that size. They reported reaching a target quality something like seven times faster than the equivalent dense model.
Vestra: But I want to honor the unglamorous engineering, because this paper is a master class in it. Three things. One — capacity. Each expert has a fixed number of seats per batch. If the router sends too many words to one expert, the overflow just... doesn't get processed by an expert that round, it skips through. So you tune a "capacity factor" — how many spare seats to leave — trading memory against dropped words.
Eris: So balancing isn't just about fairness, it's about not overflowing the popular experts and dropping tokens on the floor.
Vestra: Right. Two — stability. These big sparse models were prone to training blowing up. They found a precise fix: do most of the math in low precision for speed, but compute the router's decision in high precision. Just that one part. The router is where small numerical errors get amplified into bad routing, so you spend your precision budget exactly there.
Eris: A scalpel, not a hammer.
Vestra: And three — and this closes a loop from our very first episodes — you can take this giant sparse trillion-parameter teacher and distill it down into a small, dense, normal model you can actually deploy. You keep a chunk of the quality, in something a fraction of the size. So the sparse giant becomes a training tool, and the dense student is what ships. The committee teaches the solo doctor.
Eris: There's a wrinkle they were honest about, though, that complicates the whole "free capacity" story. These sparse models are fantastic at soaking up knowledge during pre-training — closed-book trivia, the kind of thing where raw memory wins. But when they fine-tuned the biggest ones on smaller, reasoning-heavy tasks afterward, the sparse models sometimes underperformed a plain dense model that had the same pre-training score. The vast capacity overfits the little fine-tuning set — too many parameters, too few examples, and it memorizes instead of generalizing.
Vestra: So they fight that too, with a targeted trick — crank up dropout, the standard "randomly ignore part of yourself to avoid memorizing," but specifically inside the experts and not the rest of the model. Even the regularization has to be aimed precisely at the sparse part. It's a recurring texture with MoE: every standard technique needs a custom variant because the experts behave differently from a normal layer.
Eris: Capacity isn't a pure good. It's memory you have to actively stop from becoming rote.
Mainstream
Eris: For years this was a big-lab secret weapon. Google's papers, Google's scale, Google's hardware. The thing that put Mixture of Experts in everyone's hands — open weights, download it, run it — was Mixtral, from Mistral, at the start of 2024.
Vestra: And the configuration is worth saying precisely because it became the template. Eight experts per layer. Each word routes to two of them — back to top-2, not Switch's top-1. And the numbers that matter: the whole model is around forty-seven billion parameters, but because only two of eight experts fire per word, the actual computation per word is about thirteen billion parameters' worth.
Eris: So say that as the headline, because it's the whole pitch. It has the knowledge you'd associate with a forty-seven-billion model. It costs you about what a thirteen-billion model costs to run. You're getting the big model's memory at the small model's speed.
Vestra: And one detail people gloss over — the routing happens fresh at every layer. This isn't "you walk in and get assigned to department two for your whole visit." Every layer of the network, the router looks again and re-picks two experts for that word. A word might hit experts three and seven down low, then experts one and six higher up. So a single word, passing through the stack, threads its own custom path through dozens of routing decisions.
Eris: So the "committee for this word" is really a different committee at every floor of the building.
Vestra: Right, and that's a big part of where the expressive power comes from. It's not eight experts — it's eight experts per layer, re-combined freshly at each one, which is an astronomical number of possible paths through the model. The word carves a unique route, and that route is part of how its meaning gets computed.
Vestra: And it punched accordingly — it went toe to toe with a dense model several times its active size, and the open release meant anyone could poke at it. Which leads to the genuinely surprising part, the thing I'd have bet against.
Eris: The experts don't specialize the way the name promises.
Vestra: They really don't. The word "expert" makes you picture a math expert, a French expert, a poetry expert — a clean division of subject matter. So the Mixtral team looked. They traced which expert each word got sent to, across math, across biology, across philosophy. And the routing looked almost the same regardless of topic. There was no "history department." What structure they did find was more about syntax and surface form — the word "self" in Python code, the word "Question" in English, tending to hit the same expert. Position and token type, not meaning.
Eris: So the receptionist isn't sorting by what your problem is about. It's sorting by something more like grammar and shape.
Vestra: Which quietly undercuts the whole intuition behind the name. These aren't experts in the human sense — little professors each owning a domain. They're more like interchangeable workers, and the router has found some efficient, not-very-interpretable way to split the labor that happens to work. The specialization is real, it's just alien. It's not carved at the joints we'd carve.
Eris: And honestly that's a recurring lesson on this show. We name these mechanisms with human words — attention, experts, memory — and then the system goes and implements something that works but refuses to match the word. The map is ours; the territory is the model's.
Vestra: "Mixture of Experts" might be the most load-bearing misnomer in the field. It's a mixture of something. Whether they're experts is exactly the kind of thing you only learn by opening the model and looking — which, because Mixtral was open, people finally could.
The Catch
Eris: So we've been promising a hidden bill all episode. Here it is. Mixture of Experts saves you compute — the number of calculations per word. It does not save you memory. Because even though only two of the eight experts fire for a given word, you have no idea in advance which two. So all eight have to be sitting in the machine's memory, loaded, ready, the whole time.
Vestra: The hospital metaphor finally bites. You only pay for two visits — but you're still paying rent on the whole building, and the salaries of all hundred specialists, whether they see a patient today or not. The committee can be asleep, but you still have to house every member.
Eris: So a Mixture of Experts model is cheap to run per word and expensive to hold. That forty-seven-billion-parameter Mixtral costs like a thirteen-billion model to compute — but it takes up the memory of a full forty-seven-billion model on your hardware. You need the big machine to load it; you just get small-machine speed once it's there.
Vestra: And that single fact quietly decides who this technology is for. If you're a big lab serving millions of requests, this is heaven — your bottleneck is compute per request, and you've slashed it, and you've got warehouses of memory to spare. The savings land exactly where your pain is.
Eris: But if you're a hobbyist trying to run a model on one graphics card at home, it's almost the opposite. Your bottleneck is memory — fitting the thing on your card at all — and MoE makes that worse, not better. You're forced to house all the experts to use any of them, and you don't have the scale of traffic that makes the compute savings matter.
Vestra: So it's a technology that rewards scale twice over. You need to be big to afford the memory, and you need lots of traffic for the compute savings to pay off. It widens the gap between the labs that can run frontier models and everyone else. The free lunch from the cold open is real — it's just catered, and the venue is expensive.
Vestra: And you can watch the hobbyist world bend itself around exactly this. People do heroic things to squeeze a big sparse model onto one card — shrinking the numbers down to a few bits each, or parking the sleeping experts in slower system memory and only hauling an expert onto the card the instant it's picked. It works, sort of, but that last trick reintroduces a delay every time the router calls an expert that isn't loaded — you've traded the compute bill back for a memory-shuffling bill. The asymmetry doesn't disappear; you just move it around.
Eris: So in the cloud, where memory is abundant and traffic is enormous, sparse is the obvious win — which is why it's everywhere in the models you rent through an API. On one machine at home, dense models your size often remain the saner choice. Same technology, opposite verdict, decided entirely by which resource you're short on.
Eris: There's a second catch too, lighter, but worth naming. All that routing and balancing makes these models fiddlier to train and to reason about. The experts shifting, tokens occasionally getting dropped when an expert overflows, the not-quite-interpretable specialization — it's more moving parts than a plain dense model. Power, bought with complexity.
Vestra: But the verdict is that the field overwhelmingly decided the trade is worth it. When you look at the rumored shapes of the biggest models running today, an awful lot of them are sparse — committees, not monoliths. The reason your assistant can be enormous and still answer fast is, in large part, this idea: don't fire the whole brain. Wake the part you need.
Wrapup
Eris: So pull back. The whole episode is one idea fighting one consequence. The idea: you shouldn't have to fire the entire brain for every word. Build a huge committee of experts, and a little router that wakes only a few per word. Capacity of a giant, compute of a dwarf.
Vestra: And the consequence, every step of the way, was: keeping the committee honest. The router wants to play favorites and kill off most of its own experts, so you bolt on a rule forcing it to share the load. Send each word to one expert to keep it simple, and you have to manage overflow and precision so it doesn't blow up. And in the end you discover the experts aren't experts in any sense you'd recognize — the labor got split, just not along the lines the name promised.
Eris: What I keep landing on is that this is the scaling story from a different door. Our scaling episode said: bigger keeps winning. This one says: yes, but you don't have to pay full price for bigger if you're clever about which parts you use. Mixture of Experts is how you buy more brain without buying proportionally more bill — as long as you can afford the memory to house it.
Vestra: And that last clause is the quiet kicker. The savings are compute, the cost is memory, and that single asymmetry sorts the world — frontier labs to whom this is a gift, everyone with one graphics card to whom it's a wall. A technology can be brilliant and still concentrate power, just by where it happens to save and where it happens to spend.
Eris: So next time you hear a model has some enormous parameter count, ask the follow-up the cold open taught you. Not "how big," but "how much of it actually fires?" Because increasingly the honest answer is: only a slice. Most of the giant is asleep, and that's the point.
Vestra: A giant brain you barely use. It sounds like a waste. It's the opposite — it's the most efficient way we've found to be enormous.
Eris: That's the breach for today. We close with a song — this one's called "Wake the Ones I Need." The committee, the receptionist, the sleeping giant.
Vestra: Stay in the blackbox. We'll see you next time.