speculative-decoding
Meta ships a 30B agent model that fits on one gaming GPU News
Meta released Muse Glimmer, a 30-billion-parameter open-weight agent model under the permissive Apache 2.0 license that shrinks to under 20 GB once quantized, small enough to run on a single 24 GB consumer graphics card.
The Full 2.8-Trillion-Parameter Kimi K3 Now Runs on Sixteen Desktop Boxes News
An operator has the complete Kimi K3 checkpoint running across sixteen GB10 mini-workstations wired through a single 400G switch, producing roughly 21 to 25 tokens per second for one user, on hardware with a verifiable floor around $57,200.
llama.cpp shipped DSpark for DeepSeek V4 Flash, and almost everyone called it the wrong name News
llama.cpp release b10228 merged speculative-decoding support for DeepSeek V4 Flash, but the new 0731 checkpoint embeds DSpark and ships no MTP at all, so the widely repeated "MTP support landed" advice points users at a head their model does not contain.
DeepSeek V4 Flash hits 32 tokens a second on one desktop News
A published benchmark shows DeepSeek's 284-billion-parameter V4 Flash generating 32 tokens per second entirely on one AMD Strix Halo machine, using aggressive quantization, speculative decoding and reduced expert routing.
Multi-Token Prediction: Teaching Models to Look Several Words Ahead Lesson
Multi-token prediction trains a language model to predict several upcoming tokens at once instead of just the next one, giving a richer training signal and a built-in path to faster generation.
The trick that makes AI type faster just hit the top of Hacker News News
A small model guesses ahead and a big model checks the work in parallel - and this week two efforts pushing that idea, DeepSeek's DSpark and JetSpec, lit up the front page while the community argued over whether it's truly 'lossless.'
Speculative Decoding: How AI Types Faster Without Changing a Word Lesson
A small, fast model guesses the next few words and a big, slow model checks them all in one pass - producing the exact same output, just quicker. The trick behind a lot of modern AI speedups.
vLLM recipe for DeepSeek V4 Flash Tool
An official vLLM recipe page with working launch commands for serving V4 Flash across several hardware configurations, including the flag that turns on the DSpark speculative-decoding module and the FP8 KV-cache and expert-parallel settings DeepSeek recommends.
vLLM (Kimi K3 support) Tool
The widely used open-source serving engine landed day-zero Kimi K3 support with a documented recipe, an FAQ on minimum hardware, and a K3-specific DSpark draft model for speculative decoding that roughly triples single-user throughput.
llama.cpp b10228 Tool
The release that adds DeepSeek V4 Flash's embedded DSpark speculative-decoding head, plus a converter that can split the draft tensors into a separate GGUF. Gains are workload-dependent: roughly 2x decode on large multi-GPU setups, and a measured slowdown on a 24 GB card with CPU offload.
Kimi-K3-DSpark Tool
Inferact's draft model for Kimi K3. It proposes seven tokens at a time for K3 to verify and accept or discard, and its block-diffusion backbone shares K3's attention-cache layout so no second cache format is needed. This is the component behind the 21-25 tokens per second measured on a sixteen-node GB10 cluster running the full K3 checkpoint.
JetSpec Tool
Parallel tree-drafting speculative decoding aiming for large, lossless inference speedups; project page and writeup with code, reporting up to several-times faster generation depending on the model and workload.
DeepSeek DSpark Tool
Open-source speculative-decoding implementation using parallel tree drafting to speed up text generation with no change to the model's output - the project that topped Hacker News this week. Drop-in inference speedups for self-hosted models.