memory-bandwidth
Prefill and decode: why a model's first token and its next one are completely different problems Lesson
Running a language model has two phases with opposite hardware profiles: prefill reads your whole prompt at once and saturates the chip's math units, while decode produces one token at a time and is limited almost entirely by memory bandwidth.
Apple's Mac Studio now holds 512GB of unified memory, which solves capacity and leaves speed exactly where it was News
The M5 Ultra Mac Studio configures to 512GB of unified memory at 1.2TB/s, enough to load almost any open-weight model in existence, but its memory bandwidth still sets a hard ceiling on how fast those models can generate text.
The "2x GB200 bandwidth" Chinese chip claim is a 2027 projection, and the arithmetic gives 1.67x News
A widely shared claim that a Chinese accelerator delivers twice the memory bandwidth of NVIDIA's GB200 traces to a roadmap part expected in early 2027, compared 64-at-a-time against a full NVIDIA rack, and the published numbers work out to 1.67 times at rack level while the single chip lands below a shipping GB200.
Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math Lesson
Generating text with a language model is limited by how fast weights can be moved from memory into the processor, not by how fast the processor can multiply, which is why most of a GPU sits idle during inference.