latency
Everything on Ground Truth tagged “latency” — 3 items.
Prefill and decode: why a model's first token and its next one are completely different problems Lesson
Running a language model has two phases with opposite hardware profiles: prefill reads your whole prompt at once and saturates the chip's math units, while decode produces one token at a time and is limited almost entirely by memory bandwidth.
OpenAI put its most intelligent model on Cerebras chips at 750 tokens a second News
OpenAI is previewing Ultrafast, a service tier that runs GPT-5.6 Sol on Cerebras hardware at up to 14 times the speed of standard processing and up to 750 output tokens per second.
OpenAI Rebuilt Voice So the Model Itself Decides When to Talk News
OpenAI's engineering posts on GPT-Live describe removing the separate turn detector from the audio path entirely and cutting session startup from six network round trips to one, treating a voice conversation as a live media system rather than a model feature.