Mixture of Experts: How AI Models Route Tokens
Learn how mixture-of-experts models route tokens through selected networks, how they train and run across devices, and why sparse computation does not guarantee fast or cheap inference. This audio was created with Boson AI's Higgs Audio -- https://www.boson.ai/higgs-audio
Listen on Spotify · Watch on YouTube · Pocket Casts
Many paths, a little computation
Eris: <|prosody:expressive_high|>In a 2017 study, Noam Shazeer and colleagues found that mixture-of-experts models predicted held-out text better than comparison models using similar arithmetic per text position. They tested the approach on language prediction and translation: a learned gate selected a few small networks, called experts, for each piece of text instead of running every expert.
Vestra: <|prosody:expressive_high|>But there were costs to check against that promise. Their largest language model lost quality, possibly because it used too little of the model for each input. Its measured hardware efficiency was also low; the authors thought that likely reflected how they scaled the batch size as they added GPUs.
Eris: <|emotion:surprise|>That possible explanation bothers me: having more model, but using too little of it. What is the model actually learning?
Vestra: <|prosody:expressive_high|>An ordinary neural network is a sequence of computational stages, called layers. Each layer takes numbers representing its input, transforms them, and passes new numbers on. The transformation uses parameters: numerical values adjusted during training, when the model repeatedly makes predictions and changes those values to reduce its errors.
Eris: <|prosody:expressive_high|>A language model first divides text into tokens. A token is a piece of text: sometimes a whole short word, sometimes part of a word or punctuation. The model converts each token into a list of numbers, a representation, and that representation changes as it moves through the network. Those numbers can reflect the surrounding context as well as the token itself. Does each token use all the learned weights in a layer?
Vestra: <|prosody:expressive_high|>In a conventional dense feed-forward network, a common kind of layer, the same learned transformation is applied to each token's representation. It mixes the incoming numbers using learned parameters, applies a nonlinear operation, and transforms the result again. Feed-forward means information goes through those operations in sequence. Dense here means the layer uses its full set of weights for each token, rather than selecting only a few sub-networks.
Eris: <|prosody:expressive_high|>A Transformer is an architecture that stacks such stages. One part, called attention, lets a token's representation draw on other tokens in the context; a feed-forward network then transforms the representation at each position. Different Transformer designs arrange the details differently, but that distinction is enough to locate the change.
Vestra: <|emotion:enthusiasm|>A mixture-of-experts layer can replace a feed-forward network in some of those blocks. It contains several expert networks, usually feed-forward networks with the same general shape but different learned parameters. <|prosody:expressive_high|>They are learned sub-networks, not human specialists or separate complete language models. Other components, including attention, are shared across tokens.
How a token finds its path
Eris: <|prosody:expressive_high|>The piece that makes those separate networks useful is the router, also called a gate. It's a small learned component that looks at a token's current numerical representation and scores the available experts. The same written token in a different sentence can have a different representation, so it can take a different route. How do the selected experts turn that into one result?
Vestra: <|prosody:expressive_high|>For each token, the router first calculates a score for each candidate. It selects the highest-scoring expert, or several of the highest-scoring experts. The system sends the representation to the selected networks, which each produce an output. If there are several outputs, the layer can weight them using the router's scores and add them together. That result continues through the rest of the model, including shared layers.
Eris: <|prosody:expressive_high|>Imagine four experts and a sentence containing “bank.” One occurrence is beside “river”; another is beside “loan.” Their representations may route differently because attention has incorporated context. Those routes are an illustration, not a claim that any particular expert learns riverbanks or finance.
Vestra: <|prosody:expressive_high|>Top-one routing means choosing one expert for a token. Top-k means choosing the k highest-scoring experts: top-two picks two, for example. The original sparse-gated paper explored choosing several, adding noise to scores during training and combining selected outputs. Its language models placed the expert layer between layers of an older recurrent architecture, rather than inside a Transformer.
Eris: <|emotion:enthusiasm|>Switch Transformers took a simpler route: one selected expert per token. I like how far they took that simplification. <|prosody:expressive_high|>In the paper's comparison, Switch models trained on a task of predicting missing pieces of text reached a given quality faster than the dense comparison on the same hardware, and the authors also compared them with a design that routed each token to two experts. That is a result for their training setup, not a guarantee that top-one wins every task or deployment. But with just one selected expert, how does the router itself learn?
Vestra: <|emotion:enthusiasm|>The router's score also scales the chosen expert's output in Switch. That matters for training: even though the choice of expert is discrete, <|prosody:pause|>the score provides a path for learning the router's parameters along with the experts and the rest of the model.
Teaching the router and handling overload
Eris: <|prosody:expressive_high|>Training creates a feedback loop. If the router sends most tokens to a few experts, those experts get more opportunities to improve, while neglected experts get less training. The router may then favor the popular ones even more. That can waste much of the model and leave a few devices carrying the load. That feedback loop worries me: how do you keep the neglected experts in use?
Vestra: <|prosody:expressive_high|>Training usually processes a batch, a group of token representations handled together. A balancing loss is an extra term in the training objective: it penalizes undesirable routing patterns alongside the main prediction error. Switch uses information about how many tokens go to each expert and the scores the router assigns, encouraging a more even distribution across a batch. It's a nudge, not a guarantee that each batch will be perfectly balanced.
Eris: <|sfx:sigh|>Uh, there's a tradeoff in that nudge. Strict balance constraints can hurt model performance in some setups, so the aim is enough balance for learning and execution while leaving room for the model to choose useful paths. How did another study enforce equal work?
Vestra: <|prosody:expressive_high|>The original sparse-gated study also tried a stricter alternative in some translation experiments: each expert took its highest-scoring inputs across the batch, giving every expert the same amount of work. Unlike choosing a fixed number of experts separately for each token, that can give individual inputs different numbers of experts. Because that batch-wide choice is harder with a small batch at inference, the authors trained per-expert thresholds to approximate it.
Eris: But what if an expert gets more tokens than it can process?
Vestra: <|prosody:expressive_high|>Another control is capacity: the number of tokens an expert is allowed to process in a batch. Switch sets it from the average tokens per expert and a capacity factor, which can add spare slots. Extra space handles uneven assignments, but reserved empty slots can cost memory and computation. Too little space means <|prosody:pause|>an expert can overflow.
Eris: <|prosody:expressive_high|>In the Switch implementation, an overflowing token skips that expert computation and travels onward through a residual connection, a path that carries the earlier representation around the layer. The token isn't erased from the sentence, but it misses a transformation the router selected. Other implementations can reroute or handle excess work differently. Skipping that work makes me uneasy. Did sending the token to another expert help?
Vestra: <|prosody:expressive_high|>Switch's authors tried sending overflow to another choice and didn't find an empirical benefit in their experiment. A different choice is not automatically better, so measure overflow and its effect on quality for the actual inputs, rather than assuming a capacity setting is harmless.
Smaller experts and shared work
Eris: <|prosody:expressive_high|>DeepSeekMoE explores a different choice from Switch's single expert. The authors split larger expert networks into smaller ones, then activate more of those smaller experts per token. <|emotion:enthusiasm|>That creates more possible combinations while keeping the intended expert parameter and arithmetic budget comparable in their design.
Vestra: <|emotion:enthusiasm|>They also set aside shared experts that process every token at that layer. The remaining experts are selected by the router. <|prosody:expressive_high|>The authors' hypothesis is that shared experts can learn information useful across contexts, leaving routed experts less need to duplicate it. The shared outputs and selected routed outputs contribute to the layer's result.
Eris: <|prosody:expressive_high|>In their smaller controlled comparison, DeepSeekMoE performed better across their reported tests than a baseline that routed each token to two experts. The models had the same total and active parameter counts. The tests included predicting text, answering questions and reading comprehension. When the researchers removed the shared expert or disabled preferred routed experts, performance fell; they interpret that as evidence that the components had learned different, useful contributions. Can we tell what each expert knows from that?
Vestra: <|sfx:humming|>Hmm, that's evidence for the design in those experiments, not a map of what every expert knows. The original sparse-gated paper found recurring word contexts among inputs strongly assigned to particular experts, but a routing label doesn't certify a clean human-readable specialty. Different experts can overlap, and the model learns the routes and the expert transformations together.
Eris: <|prosody:expressive_high|>I can see why you resist giving each expert a neat job title. Smaller experts also bring an engineering limit. DeepSeek's authors avoided making experts finer still in one larger configuration because very small computations could use hardware less efficiently. More possible routes are useful only if the model learns and executes them well.
The cost of moving tokens
Eris: <|prosody:expressive_high|>The word sparse describes the computation for an individual token. Total parameters count the whole model, including every expert. Active parameters count the parameters used on a particular pass, including shared parts and selected experts. A large gap between those counts is possible, but all those alternatives still have to be stored or made available when needed. So how much comfort should I take from that smaller active count when memory is tight?
Vestra: <|prosody:expressive_high|>That means a sparse model is not necessarily a small download or a low-memory model. Loading all experts into accelerator memory can be expensive; moving them in and out of other memory can introduce delays. Training also needs space for intermediate values and the information used to update parameters.
Eris: <|prosody:expressive_high|>An accelerator is a processor such as a GPU designed to do large amounts of numerical work. When the experts don't fit on one accelerator, or when more parallel work is useful, a system can spread them across several. That's expert parallelism.
Vestra: <|emotion:enthusiasm|>Now the route becomes physical. The router decides which experts need each token representation. The system dispatches those representations to the devices holding those experts; the experts compute their results; then the results travel back to be combined and passed onward. <|prosody:expressive_high|>During training, information needed to adjust the model also has to travel backward through that computation.
Eris: <|emotion:surprise|>That round trip is where <|prosody:pause|>a theoretical saving can disappear. <|prosody:expressive_high|>Even if fewer arithmetic operations are needed, devices can wait on communication, uneven workloads or very small groups of tokens that don't use their hardware efficiently. Latency means the time a request takes; throughput means how much work the system finishes over a period. An arrangement that helps one may not help the other. Which of these choices can a developer actually control?
Vestra: <|prosody:expressive_high|>DeepSpeed's documented MoE interface lets a developer set the number of experts and the size of an expert-parallel group, whose participating devices distribute those experts. Its tutorial also describes combining expert parallelism with data parallelism, where groups handle different batches, and with other ways of dividing model work or using CPU memory. Those are implementation options in DeepSpeed, not requirements of every mixture-of-experts model.
What to measure
Eris: <|prosody:expressive_high|>If you're choosing a model, the expert count alone tells you neither how capable it is nor how quickly it will answer. Start by asking for total and active parameter counts, what each count includes, how many experts are selected, and whether some experts are always shared. What would you check once those counts look promising?
Vestra: <|prosody:expressive_high|>Then check the routing under your own inputs: how uneven is the work across experts and devices, how often does capacity overflow, and what happens to an overflowing token? Measure memory and communication on the intended hardware. Test response latency and throughput at realistic request sizes and levels of simultaneous traffic, including the cost of loading or moving weights where relevant.
Eris: <|emotion:determination|>And compare quality on the tasks you actually need, against a suitable dense model as well as other expert designs. <|prosody:expressive_high|>Switch found that improvements at its initial text-prediction task didn't always translate equally to later tasks. A model can look efficient on one measurement and still be the wrong choice for your workload.
Vestra: <|prosody:expressive_high|>That task comparison matters to me more than the expert count. The practical question is whether the learned routing <|prosody:pause|>earns its costs: storage for the possible paths, training to keep them useful, and the movement needed to reach them. Only a measurement of the complete system can answer that for a particular job.
Follow the research
Eris: <|prosody:expressive_high|>For more AI reporting, visit Ground Truth at groundtruth.day. <|emotion:enthusiasm|>Follow AI in Detail for more explanations like this one. What would you measure first when choosing an expert model? Let us know in the comments.
Vestra: <|emotion:contentment|>Thanks for listening.