serving
Prefill and decode: why a model's first token and its next one are completely different problems Lesson
Running a language model has two phases with opposite hardware profiles: prefill reads your whole prompt at once and saturates the chip's math units, while decode produces one token at a time and is limited almost entirely by memory bandwidth.
Inference cost and token economics: why output tokens cost more than input Lesson
Model providers charge separately for the tokens you send and the tokens the model writes, and output is typically three to five times more expensive. The reason is architectural: input is processed in one parallel pass, while every output token requires its own full pass through the model.
DFlash 2 decodes 3.4 times faster, and published the table showing where that stops being true News
Inco AI released DFlash 2, a block-diffusion drafter for speculative decoding that reaches 3.43 times the throughput of ordinary decoding on a single request, and published benchmark tables showing the advantage shrinking to almost nothing under heavy concurrency.
vLLM v0.23.0 Tool
The widely-used open engine for serving language models fast and cheaply. The latest release adds smarter memory handling for long conversations and faster GPU execution.
vLLM recipe for DeepSeek V4 Flash Tool
An official vLLM recipe page with working launch commands for serving V4 Flash across several hardware configurations, including the flag that turns on the DSpark speculative-decoding module and the FP8 KV-cache and expert-parallel settings DeepSeek recommends.
vLLM DeepSeek-V4 support Tool
vLLM shipped serving support for DeepSeek-V4's compressed long-context attention, including hybrid KV-cache management, multiple cache page sizes, kernel fusion and multi-stream partitioning. The engineering post documents the recipe and the hardware it assumes.
vLLM (Kimi K3 support) Tool
The widely used open-source serving engine landed day-zero Kimi K3 support with a documented recipe, an FAQ on minimum hardware, and a K3-specific DSpark draft model for speculative decoding that roughly triples single-user throughput.
vLLM Tool
The popular open engine for serving AI models fast and efficiently when you need to handle real traffic.
SGLang v0.5.13 Tool
A high-performance open serving engine for language models. The new version turns on faster 'guess-ahead' decoding by default and trims scheduling overhead for quicker responses.
SGLang K2 Horizon Cookbook Tool
IFM's validated serving configuration for the K2 Horizon family, with measured H200 latency and throughput for every model size. Covers the tensor-parallel setup and the router numerics override that preserves checkpoint behaviour -- the difference between the model running and the model running correctly.
SGLang (Kimi K3 cookbook) Tool
Alternative open-source serving engine with day-zero K3 support and a step-by-step deployment cookbook. Its writeup documents how prefix caching, paging and prefill/decode disaggregation were rebuilt to handle K3's mix of recurrent and key-value state.
SGLang Tool
The other serving stack DeepSeek's official 0731 model card documents as supporting DSpark directly, alongside the recommended FP8 key-value cache and FP4 indexer cache configuration for V4 Flash.
Ling-3.0-flash Tool
inclusionAI's 124B mixture-of-experts model with about 5.1B parameters activated per token. Sparse routing genuinely cuts per-token compute, but this is a server-class artifact, not a laptop one: the BF16 repository is roughly 255GB and the official serving path calls for custom SGLang or vLLM forks with tensor parallelism across four GPUs.
DFlash 2 (Qwen3.8-27B drafter) Tool
A drop-in block-diffusion drafter for speculative decoding on Qwen3.8-27B, with documented launch commands for SGLang and vLLM. Output is provably identical to the target model; throughput gains reach 3.4x on single requests and shrink under heavy concurrency.