throughput
Everything on Ground Truth tagged “throughput” — 2 items.
Prefill and decode: why a model's first token and its next one are completely different problems Lesson
Running a language model has two phases with opposite hardware profiles: prefill reads your whole prompt at once and saturates the chip's math units, while decode produces one token at a time and is limited almost entirely by memory bandwidth.
Kimi-K3-DSpark Tool
Inferact's draft model for Kimi K3. It proposes seven tokens at a time for K3 to verify and accept or discard, and its block-diffusion backbone shares K3's attention-cache layout so no second cache format is needed. This is the component behind the 21-25 tokens per second measured on a sixteen-node GB10 cluster running the full K3 checkpoint.