memory-bandwidth
Everything on Ground Truth tagged “memory-bandwidth” — 1 item.
Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math Lesson
Generating text with a language model is limited by how fast weights can be moved from memory into the processor, not by how fast the processor can multiply, which is why most of a GPU sits idle during inference.