Learn · Intermediate

Distributed training: how one model gets split across thousands of chips

Training a frontier AI model requires splitting one model across thousands of chips, because no single chip comes close to holding it. There are four fundamentally different ways to make that split -- by data, by layer, by tensor, and by expert -- and real training runs combine several at once. Getting the combination right is the difference between a cluster running near its theoretical peak and one spending most of its time waiting on the network. This is the least glamorous part of modern AI and one of the most decisive.

Why one chip is never enough

Start with the memory arithmetic, because it is more brutal than most people expect. A model's parameters are only the beginning. During training you also need the gradients -- one number per parameter, telling you which direction to adjust -- and the optimizer state, which for the standard AdamW optimizer is two more numbers per parameter. On top of that sit the activations: every intermediate value computed on the way forward, kept around because backpropagation needs them on the way back.

A rough rule is that training needs something like six to eight times the memory of the raw weights. A model with a hundred billion parameters, stored at two bytes each, is 200 gigabytes of weights and well over a terabyte to actually train. The largest single accelerators today offer under 200 gigabytes. There is no version of this that fits on one chip.

The four splits

Data parallelism is the simplest and the oldest. Every device gets a complete copy of the model and a different slice of the training batch. Each computes its own gradients, then all devices average them together so every copy stays identical. The communication step is one big all-reduce -- everyone contributes, everyone receives the sum. It scales throughput beautifully and saves no memory at all, since every device still holds the whole model.

Pipeline parallelism cuts the model by layer. Device one holds layers 1 through 10, device two holds 11 through 20, and so on. Data flows through like an assembly line. The obvious problem is that while device one works, devices two through eight sit idle -- a waste the GPipe paper named the "bubble." The fix is to chop each batch into micro-batches and keep several in flight at once, so every stage always has something to work on. It never gets the bubble to zero, only small.

Tensor parallelism cuts inside a layer. A single matrix multiplication is split column-wise or row-wise across several devices, each computing part of the result, with the pieces stitched back together. Introduced at scale by Megatron-LM, it lets a layer too big for one chip run across several -- at the cost of communication inside every layer, several times per forward pass. That traffic is so heavy that tensor parallelism is normally confined to devices inside a single server, connected by the fastest links available.

Expert parallelism is the newest and applies to mixture-of-experts models. Each device holds a few of the model's expert sub-networks. Because a router sends each token to only a couple of experts, tokens must be shipped to wherever their experts live, processed, and shipped back. The communication pattern is an all-to-all -- every device sending a different piece to every other device -- which GShard established as the standard approach.

Combining them, and why it is hard

Real runs use several at once, often called 3D or 4D parallelism: tensor parallelism inside a server, pipeline parallelism across servers, data parallelism across the whole cluster, and expert parallelism layered in for sparse models. The Megatron GPU-cluster paper is the canonical study of how these compose.

The reason there is no universal answer is that each strategy trades memory against a different kind of communication, and the machine you are on has different speeds for each hop. Devices inside one server might talk at terabytes per second; servers within a rack, much slower; racks across a data center, slower still. A configuration tuned for one topology can spend most of its time waiting on another.

Think of it like organizing a large kitchen. You can run identical stations each cooking full meals (data parallel), or an assembly line where each station adds one component (pipeline), or split a single hard dish across several pairs of hands (tensor), or route each order to whichever specialist handles it (expert). Which arrangement wins depends entirely on how far apart the stations are and how much shouting is needed between them.

Where the memory pressure actually goes

One complementary technique deserves mention because it is nearly free. ZeRO, from Microsoft's DeepSpeed team, observed that in plain data parallelism every device holds an identical copy of the optimizer state, gradients, and parameters -- pure redundancy. ZeRO shards those across the data-parallel devices and gathers the pieces only when needed. It delivers large memory savings without changing the model's mathematics at all, which is why it became a default.

Why this matters beyond the data center

Parallelism is where hardware independence is actually decided. When a team ports training to a non-Nvidia accelerator, the framework switch is the easy part. What has to be rebuilt is the collective-communication library implementing all-reduce and all-to-all, the scheduling that overlaps that communication with computation so chips are not idle, and the hand-tuned low-level kernels for the operations that dominate runtime. Recent Huawei Ascend training reports make exactly this point: the work list is redesign parallelism, reschedule collectives, convert checkpoints, and rewrite kernels by hand.

That is the real bill for leaving a mature software ecosystem -- and it is why announcements about chip alternatives should be read as claims about software maturity, not silicon.

Ground Truth. (2026, July 23). Distributed training: how one model gets split across thousands of chips. Ground Truth. https://groundtruth.day/learn/distributed-training-parallelism.html

@misc{groundtruth:distributed-training-parallelism,
  title  = {Distributed training: how one model gets split across thousands of chips},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/distributed-training-parallelism.html}
}

Topics: training · distributed · infrastructure · mixture-of-experts · fundamentals