inference
Beam search: keeping several drafts alive to find a likelier sentence Lesson
Beam search is a decoding method that keeps the few highest-scoring partial outputs at every step instead of committing to the single likeliest word, which finds more probable sentences than greedy decoding and exposes why the most probable text is often not the best text.
DeepSeek ships V4.1 Flash as a 510 GB open-weights download News
DeepSeek released V4.1 Flash under an MIT licence on 10 September 2026: a 552-billion-parameter backbone plus a separate 196-billion-parameter memory module, split into an encoder and a decoder so that reading text costs half as much compute as writing it.
Continuous batching and paged attention: how one GPU serves a thousand people at once Lesson
Continuous batching and paged attention are the two scheduling tricks that let a single GPU serve many users at once instead of one at a time, and together they are responsible for most of the collapse in the price of running a language model.
Multi-head latent attention: compressing the memory that inference actually runs out of Lesson
Multi-head latent attention compresses the key-value cache that language models must hold in GPU memory during generation, projecting keys and values into a small shared latent vector instead of storing them per head. DeepSeek introduced it in DeepSeek-V2, reporting a cache reduction of more than 90% with quality matching full attention.
Cerebras is serving an open 27B model at 1,500 tokens a second, and the free tier caps it exactly News
Cerebras now serves Qwen 3.8 27B at roughly 1,500 output tokens per second, but its own rate-limit page caps free-tier users at 90,000 tokens per minute -- almost precisely the model's raw output rate -- so the headline speed only becomes usable on the paid tier.
Test-Time Training Lesson
Test-time training is the practice of updating a model's actual weights on the specific problem in front of it, at inference, rather than only running a forward pass -- turning each test example into a tiny training run.
The small-model argument hit the front page News
Segment co-founder Calvin French-Owen argued that cheap fast models have crossed a usefulness threshold, pricing a personalized-news task he once ran for about a dollar at roughly ten cents, and the essay drew 499 points on Hacker News.
fal post-trained MiniMax H3 and kept the weights News
Inference company fal released H3 Max, a post-trained version of the open-weight MiniMax H3 video model that renders a five-second 768p clip in under three seconds -- available only as a hosted API, with no weights published.
GLM-5.3-Flash was Ox Alpha, and it ran on Chinese chips News
Z.ai released GLM-5.3-Flash under an MIT licence and confirmed it is the anonymous \u201cOx Alpha\u201d model that topped OpenRouter for a week -- served, the company says, entirely on a cluster of Chinese AI accelerators at per-token cost comparable to NVIDIA hardware.
Prefill and decode: why a model's first token and its next one are completely different problems Lesson
Running a language model has two phases with opposite hardware profiles: prefill reads your whole prompt at once and saturates the chip's math units, while decode produces one token at a time and is limited almost entirely by memory bandwidth.
OpenAI publishes first Jalapeno results, claiming up to 1.9x more work per watt than the systems it tested against News
OpenAI released measured results for Jalapeno, its Broadcom-co-designed inference chip, reporting 1.5-1.9x more AI work per watt, 1.7-3.6x lower latency, and 2.1-4.1x higher performance on interactive workloads, with kernels its own model wrote.
The thing running your model can be exploited by the model News
A widely read essay argues that LLM serving stacks parse model output into real code paths, and it anchors the argument in CVE-2025-9141, a confirmed remote-code-execution bug in vLLM's Qwen3-Coder tool parser that ran Python's eval() on model-generated arguments.
OpenAI cut Sol's price, and OpenRouter cut it again News
OpenAI dropped GPT-5.6 Sol to $4 per million input tokens and $20 per million output on August 21, 2026, a 33 percent cut on output, and OpenRouter is separately listing the same model from OpenAI at half that.
LLMs are less resilient to bit flips than accuracy suggests News
A supercomputing-conference study that injected more than 13 million simulated hardware faults into language model inference found that benchmark accuracy barely moves while the quality of generated text degrades badly, and that 4-bit quantized models are more robust than full-precision ones.
Inference cost and token economics: why output tokens cost more than input Lesson
Model providers charge separately for the tokens you send and the tokens the model writes, and output is typically three to five times more expensive. The reason is architectural: input is processed in one parallel pass, while every output token requires its own full pass through the model.
Self-hosting a frontier open model now takes a whole node News
Engineers at the Belgian research institute imec benchmarked open-weight coding models against commercial APIs and found the best open model needs an eight-GPU B300 node to run at all, serving only 16 concurrent sessions at roughly eight times slower than a hosted frontier agent.
Why temperature zero is not deterministic Lesson
Setting temperature to zero makes a model always pick its highest-scoring next token, but it does not make the model return the same answer twice, because batching, floating-point arithmetic, and expert routing change the scores themselves between runs.
Stripe is buying the company that keeps score on every model News
Stripe announced on August 19 that it has agreed to acquire OpenRouter, the gateway that routes developer traffic across more than 400 models from over 80 providers and publishes the industry's most-watched model usage rankings.
Etched raised 700 million dollars and shipped its first rack to Jane Street News
Etched announced on August 18 that it shipped its first inference rack to Jane Street and raised 700 million dollars at a 21 billion dollar valuation, betting that frontier inference belongs on hardware co-designed for it rather than on general-purpose accelerators.
DeepSeek is selling a checkpoint it has not published News
DeepSeek's API now serves a model version named DeepSeek-V4-Pro-0813 and at least five commercial hosts resell it by that exact name, but the company has not published a matching dated weights page, and none of the resellers undercuts DeepSeek's own price.
A 753 billion parameter model ran on a single workstation GPU News
A serving system called FreeToken reports running frontier-scale sparse models on ordinary personal hardware, including a 753 billion parameter model on one workstation GPU and a 35 billion parameter model on a laptop with 8 gigabytes of video memory.
DFlash 2 decodes 3.4 times faster, and published the table showing where that stops being true News
Inco AI released DFlash 2, a block-diffusion drafter for speculative decoding that reaches 3.43 times the throughput of ordinary decoding on a single request, and published benchmark tables showing the advantage shrinking to almost nothing under heavy concurrency.
Models are being trained to give up News
New work targets the compute burned when a model produces confident, plausible reasoning on problems it cannot solve, with one method training models to refuse instead of reasoning and another cutting total token use by up to 68.5 percent by reallocating compute mid-thought.
OpenRouter now picks your model by what everyone else is paying for News
OpenRouter rebuilt its automatic model router around aggregate spending data from the past seven days, replacing hand-tuned rules with a crowd signal drawn from more than 55 trillion tokens a week.
OpenAI put its most intelligent model on Cerebras chips at 750 tokens a second News
OpenAI is previewing Ultrafast, a service tier that runs GPT-5.6 Sol on Cerebras hardware at up to 14 times the speed of standard processing and up to 750 output tokens per second.
DeepSeek starts charging rush-hour prices on August 17 News
DeepSeek is replacing flat API pricing with peak and off-peak rates on August 17, and the steepest change hits cached input on its Pro model, which goes up twelvefold during Beijing business hours.
A stronger model built a wrapper that nearly doubled a weaker one's score News
Researchers had a strong model design inference-time scaffolding for weaker models, lifting their average score on four reasoning benchmarks from 0.49 to 0.91 without changing a single parameter.
DeepSeek's new open model is 1.6 trillion parameters and runs 49 billion of them per token News
DeepSeek published DeepSeek-V4-Pro on Hugging Face with 1.6 trillion total parameters, 49 billion activated per token, and a one-million-token context window, making it the largest openly downloadable model of the current frontier wave.
NVIDIA built a 30B model for the boring half of agent work News
NVIDIA released Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts model that activates only 3 billion parameters per token and is aimed at the tool calls and validation steps that consume most of a long-running agent's time.
The Full 2.8-Trillion-Parameter Kimi K3 Now Runs on Sixteen Desktop Boxes News
An operator has the complete Kimi K3 checkpoint running across sixteen GB10 mini-workstations wired through a single 400G switch, producing roughly 21 to 25 tokens per second for one user, on hardware with a verifiable floor around $57,200.
High Bandwidth Flash Became a Spec Today, Not a Product You Can Buy News
SK hynix and Sandisk published the first standard for High Bandwidth Flash at FMS 2026, defining a NAND memory tier of up to 512GB per stack with a top bandwidth grade near three terabytes a second - with no named accelerator, price or availability date.
A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type News
An open pull request to llama.cpp tracks which mixture-of-experts submodels get used most during inference and promotes them to GPU memory on the fly, roughly doubling decode speed on an 8GB card in the author's own tests - while slowing other models down.
The Cheap 284B Rig Is Really 768GB of Server Memory News
A builder running DeepSeek V4-Flash at 33 tokens a second on two RTX 3090s is holding about 6.6GB of weights per card and roughly 170GB per instance in system memory on a four-socket enterprise server, which is where the model actually lives.
llama.cpp shipped DSpark for DeepSeek V4 Flash, and almost everyone called it the wrong name News
llama.cpp release b10228 merged speculative-decoding support for DeepSeek V4 Flash, but the new 0731 checkpoint embeds DSpark and ships no MTP at all, so the widely repeated "MTP support landed" advice points users at a head their model does not contain.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
Offloading and streaming: running a model bigger than your memory Lesson
Offloading and streaming let a machine run a model far larger than its memory by keeping only the parts needed right now in fast memory and fetching the rest from system RAM or disk on demand, trading speed for capacity.
Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token News
A hand-written C engine generates text with Moonshot's 2.8-trillion-parameter Kimi K3 using a peak of 8.24 gigabytes of RAM and no GPU, by reading the model's four-bit experts directly off disk, at a rate of roughly one token every 33 seconds.
DeepSeek's low effort setting writes more than its high setting, because the dial is just a prompt News
A reproduction across both a local copy and DeepSeek's hosted API found V4 Flash consuming substantially more tokens on its low reasoning-effort setting than on high, and the released encoder explains why: low injects no instruction at all while high prepends a paragraph demanding exhaustive deliberation.
A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill News
An open-source engine called Mference runs DeepSeek V4 Flash on a 24 GB Mac with an effective memory footprint of about 3 gigabytes by streaming each token's experts off the SSD, but the checkpoint still occupies 90 to 98 gigabytes of disk and the test ran at a 4,000-token context.
OpenAI cut its cheapest model's price 80%, and credits one of its own models for making it possible News
OpenAI dropped GPT-5.6 Luna's API price by 80% and Terra's by 20% effective July 30, and says its Sol model autonomously rewrote production kernels that cut the cost of serving the model by 20%.
OpenAI says GPT-5.6 Sol autonomously rewrote the code that serves it, cutting serving costs 20% News
OpenAI published an engineering account on July 29 saying GPT-5.6 Sol, working through Codex, autonomously rewrote its production GPU kernels and redesigned its own draft model, contributing to a 20% cut in end-to-end serving cost and a 15% gain in token-generation efficiency.
NVIDIA shipped a drop-in kernel that nearly halves video generation time News
NVIDIA released code on July 28 for Sol-Attn, an attention kernel that decides which parts of a long video to compute exactly while approximating the rest inside a single pass, reporting up to 2.1 times faster video generation with no retraining and no weight changes.
A frozen 12B model answers already-solved problems at zero generation tokens News
A technical report describes a 12-billion-parameter model whose weights never change but which answers new instances of nine previously solved problem families with no generated tokens at all, scoring 180 out of 180 by executing verified stored procedures instead of reasoning again.
Chinese open models passed US models in OpenRouter token share News
OpenRouter's own usage data shows Chinese models overtaking US models in token volume in early June, with DeepSeek roughly doubling its share to 18% - driven by token-hungry agent workloads routing to the cheapest capable endpoint.
Kimi K3 is downloadable, but the floor to run it is eight datacenter GPUs News
Kimi K3's 1.56-terabyte checkpoint needs a single eight-GPU B300 or MI355X node as its practical minimum, and no version of llama.cpp can load it today, so open weights currently mean operator-scale rather than local.
Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math Lesson
Generating text with a language model is limited by how fast weights can be moved from memory into the processor, not by how fast the processor can multiply, which is why most of a GPU sits idle during inference.
Model Routing and Cascades: Spending Frontier Money Only Where It Buys Something Lesson
Model routing sends each request to the cheapest model that can handle it, and cascades escalate to a stronger model only when a cheaper one is not confident, which cuts serving costs substantially because most production traffic is easy.
AMD and Cerebras split AI inference across two different chips News
AMD and Cerebras announced a joint inference offering on July 23 in which AMD's Helios racks process the prompt and Cerebras's wafer-scale engine generates the tokens, claiming up to five times the tokens per watt of a Cerebras-only setup.
GigaToken Claims a 1,000x Tokenizer Speedup, But the Real Serving Win Is Single-Digit News
A new tokenizer called GigaToken reports roughly 1,000 times the offline throughput of Hugging Face Tokenizers, but its benchmark is not apples-to-apples and its real inference payoff is a single-digit latency improvement.
Google's two opposite bets: a Gemini-specialized chip and an EU order to open Android AI News
Google is reportedly designing a server chip called Frozen v2 that hardwires Gemini's architecture for six-to-ten times more tokens per watt, even as the European Commission adopted binding measures forcing Android to open eleven AI capabilities to rival assistants, making Google simultaneously bet on locking Gemini into silicon and being forced to unlock Gemini's Android advantages.
Gemini 3.6 Flash: Google ships a faster worker, not a bigger brain News
Google released Gemini 3.6 Flash into general availability, and independent benchmarks show it streams output nearly twice as fast as 3.5 Flash and costs less per task while scoring the same on a leading intelligence index, though it still takes a conspicuous 11-plus seconds to start responding.
Google Falls Off One Leaderboard's Top 15, as a Report Describes a Gemini-Specific Chip News
Google dropped out of the top 15 on LLM Stats' composite leaderboard while remaining its fastest model, and Reuters separately reported an unannounced Gemini-specific inference chip.
AMD Absorbs FastFlowLM Team to Build GPU-Free NPU Inference News
AMD announced on July 17, 2026 that the FastFlowLM team has joined its Artificial Intelligence Group to build out an NPU-first, GPU-free local inference stack for Ryzen AI laptops.
Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies Lesson
Prompt caching lets an AI provider store the processed form of a repeated chunk of text -- like a long system prompt -- so it can be reused across requests at a fraction of the cost, instead of being re-processed every time.
DeepSeek is designing its own AI chip -- and raising outside money for the first time News
Chinese AI startup DeepSeek is developing its own chip aimed at running trained models rather than training them, and is simultaneously raising its first-ever outside capital -- about $7 billion at a $52-59 billion valuation.
New methods page an AI's memory in and out like an operating system News
Two new techniques treat a language model's long-context memory like an operating system's memory hierarchy - keeping coarse summaries on the GPU and paging compressed detail out to the CPU - with one, SeKV, cutting GPU memory use by 53% at 128,000 tokens while avoiding the errors that plague simpler methods.
A startup router is giving away 100 million tokens of Kimi, MiniMax and GLM News
API aggregator Dahl Inference is handing out 100 million free tokens across top open-weight Chinese models like Kimi K2.6 and MiniMax M2.7 - not a price cut from the labs themselves, but a router burning money to win users amid a glut of cheap compute.
A $4-per-million open model is coming for the frontier's 90% margin News
GLM-5.2, an open-weights model priced at under a fifth of Opus, scores as the top open model and 4th overall - and a widely-shared essay argues it is the first real threat to frontier labs' ~90% inference margins.
Program-as-Weights compiles a plain-English spec into a tiny model you run on a laptop News
A new method called Program-as-Weights uses a 4-billion-parameter 'compiler' to turn a natural-language task description into a small weight file that a frozen 0.6B model runs, matching a 32B model's quality while using about one-fiftieth the memory and running at 30 tokens a second on a MacBook.
Test-Time Compute: Spending More Thinking at the Moment You Ask Lesson
Test-time compute is the idea of making an AI smarter not by retraining it, but by letting it do more work the moment you ask a question: think longer, or generate many answers and pick one.
GPT-5.5 Codex Keeps Cutting Its Own Reasoning Off at Exactly 516 Tokens News
A GitHub analysis of 390,195 coding-session responses found GPT-5.5 disproportionately cuts off its own reasoning at exactly 516 tokens, a pattern likely caused by a batching bug rather than an intentional change.
Temperature and top-p: how an AI actually picks its next word Lesson
A language model does not know its next word - it produces a list of odds and then rolls dice. The rules of that dice roll are why the same prompt gives you a boring answer one day and a wild one the next.
Ollama nearly doubles Gemma's speed on Macs by guessing ahead News
A free local-AI tool now runs Google's Gemma model far faster on Apple computers using a trick where a small model drafts words and the big one checks them in bulk.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
The trick that makes AI type faster just hit the top of Hacker News News
A small model guesses ahead and a big model checks the work in parallel - and this week two efforts pushing that idea, DeepSeek's DSpark and JetSpec, lit up the front page while the community argued over whether it's truly 'lossless.'
Speculative Decoding: How AI Types Faster Without Changing a Word Lesson
A small, fast model guesses the next few words and a big, slow model checks them all in one pass - producing the exact same output, just quicker. The trick behind a lot of modern AI speedups.
Quantization: Shrinking AI Models to Run on Modest Hardware Lesson
Storing a model's numbers with less precision - 8, 4, or even fewer bits instead of 16 - makes it dramatically smaller and faster, often with almost no loss in quality. It's why big models can run on a laptop or a single GPU.
Frontier AI is getting more expensive while open models keep getting cheaper News
Closed frontier models are raising prices and tightening access just as Chinese open-weight models slash theirs, a structural reversal with big consequences for who builds with AI.
Chain-of-thought: why making an AI think out loud makes it smarter Lesson
Asking a model to work through a problem step by step, instead of blurting an answer, dramatically improves it on hard tasks. Here is why that simple trick works, what it really buys the model, and where it backfires.
Training vs inference: the two very different jobs inside every AI Lesson
Why building an AI model and using it are separate worlds with separate costs, and why that split explains custom chips, model prices, and where the real money in AI actually goes.
OpenAI designs its own chip to run its models News
With Broadcom, OpenAI unveiled a custom chip built for one job: serving its AI models cheaply.
Two labs race to make AI write whole paragraphs at once instead of word by word News
Diffusion text models generate in parallel blocks rather than left to right; Google's open DiffusionGemma and Inception's Mercury 2 are now in a head-to-head over speed.
Suddenly, downloadable AI models look like an insurance policy News
With a top hosted model pulled overnight, a flood of powerful open models you can run yourself -- and run fast -- is being reframed from hobby to risk management.
vLLM v0.23.0 Tool
The widely-used open engine for serving language models fast and cheaply. The latest release adds smarter memory handling for long conversations and faster GPU execution.
vLLM recipe for DeepSeek V4 Flash Tool
An official vLLM recipe page with working launch commands for serving V4 Flash across several hardware configurations, including the flag that turns on the DSpark speculative-decoding module and the FP8 KV-cache and expert-parallel settings DeepSeek recommends.
vLLM Skills Tool
Packaged skills for working with vLLM, published by the vLLM project itself, in the same portable Agent Skills format. Useful if you want an agent that can actually configure and debug a vLLM deployment rather than guessing at flags.
vLLM DeepSeek-V4 support Tool
vLLM shipped serving support for DeepSeek-V4's compressed long-context attention, including hybrid KV-cache management, multiple cache page sizes, kernel fusion and multi-stream partitioning. The engineering post documents the recipe and the hardware it assumes.
vLLM (Kimi K3 support) Tool
The widely used open-source serving engine landed day-zero Kimi K3 support with a documented recipe, an FAQ on minimum hardware, and a K3-specific DSpark draft model for speculative decoding that roughly triples single-user throughput.
llama.cpp b10228 Tool
The release that adds DeepSeek V4 Flash's embedded DSpark speculative-decoding head, plus a converter that can split the draft tensors into a separate GGUF. Gains are workload-dependent: roughly 2x decode on large multi-GPU setups, and a measured slowdown on a 24 GB card with CPU offload.
llama.cpp Tool
The lean, fast engine that makes big models run on ordinary laptops; powers much of the local-AI ecosystem.
bitnet.cpp Tool
Microsoft's official inference framework for 1.58-bit ternary language models, built on llama.cpp with optimized CPU and GPU kernels for running very heavily compressed models on ordinary hardware.
SwiftLM Tool
An MLX-based runtime for Apple silicon whose --stream-experts mode reads mixture-of-experts weights straight off an NVMe SSD, letting a machine run models several times larger than its RAM.
Sol-Attn (Sol-Engine) Tool
NVIDIA's drop-in sparse attention kernel for long-video diffusion transformers, released July 28 for HunyuanVideo-13B and Wan2.1-T2V-14B. Screens compressed key/value blocks inside a single online-softmax pass, so exact attention goes where it matters and skipped blocks get an approximate correction. Training-free, no weight changes, reported up to 2.1x for generation and 2.3x for editing. The repo marks end-to-end re-benchmarks for the two integrated pipelines as pending.
SGLang v0.5.13 Tool
A high-performance open serving engine for language models. The new version turns on faster 'guess-ahead' decoding by default and trims scheduling overhead for quicker responses.
SGLang K2 Horizon Cookbook Tool
IFM's validated serving configuration for the K2 Horizon family, with measured H200 latency and throughput for every model size. Covers the tensor-parallel setup and the router numerics override that preserves checkpoint behaviour -- the difference between the model running and the model running correctly.
SGLang (Kimi K3 cookbook) Tool
Alternative open-source serving engine with day-zero K3 support and a step-by-step deployment cookbook. Its writeup documents how prefix caching, paging and prefill/decode disaggregation were rebuilt to handle K3's mix of recurrent and key-value state.
SGLang Tool
The other serving stack DeepSeek's official 0731 model card documents as supporting DSpark directly, alongside the recommended FP8 key-value cache and FP4 indexer cache configuration for V4 Flash.
Program-as-Weights Tool
Turns a plain-English task spec into a small weight file that a frozen 0.6B model runs locally -- matching a 32B model's quality at roughly one-fiftieth the memory and about 30 tokens/sec on a MacBook M3. Open repo and site for compiling cheap, offline 'fuzzy' text programs.
OpenRouter Auto router Tool
Single endpoint that picks a model per request using the past seven days of aggregate platform spend on similar tasks, with a cost_tier parameter to set how much you want to spend and account-level guardrails respected.
OpenRouter Auto Exacto Tool
OpenRouter's provider-routing system that repeatedly evaluates provider telemetry and benchmark behavior, then deranks statistical outliers.
Ollama 0.31 Tool
Run open models on your own computer; the new version nearly doubles Gemma's speed on Apple Silicon using multi-token prediction, on by default.
NVIDIA Nemotron 3.5 Lightning Tool
A 30B mixture-of-experts model with only 3B parameters active per token, trained for the high-volume half of agent work: tool calls, result validation and subagent delegation. NVFP4 and BF16 checkpoints, with weights, training data and recipes released under a permissive licence.
NInfer Tool
A focused inference engine that runs Qwen 3.6 models on one RTX 5090 with a 262,000-token context using an INT8 key-value cache, reporting roughly 188 tokens per second at 250,000 tokens of context. Methodology, seeds and limits are published openly.
Modular MAX + Mojo Tool
A programming language (Mojo) and compiler/runtime (MAX) for running AI models efficiently across different hardware instead of being locked to one chip vendor; now being acquired by Qualcomm but still openly available to developers.
Mference Tool
Runs DeepSeek V4 Flash on Apple silicon by keeping the shared core, attention and cache resident while streaming each token's routed experts off the SSD. Publishes an unusually honest memory budget: about 3 GB working set, 90 to 98 GB on disk, tested at a 4,000-token context on a 24 GB Mac, with no quality parity test yet.
LvLLM Tool
A community inference runtime specialised in hybrid CPU and GPU execution of mixture-of-experts models, with NUMA-aware scheduling, expert weight management, and MXFP4 quantization kernels. Its DeepSeek V4 build publishes a working dual-RTX-3090 launch configuration at 22K context.
Kimi-K3-DSpark Tool
Inferact's draft model for Kimi K3. It proposes seven tokens at a time for K3 to verify and accept or discard, and its block-diffusion backbone shares K3's attention-cache layout so no second cache format is needed. This is the component behind the 21-25 tokens per second measured on a sixteen-node GB10 cluster running the full K3 checkpoint.
JetSpec Tool
Parallel tree-drafting speculative decoding aiming for large, lossless inference speedups; project page and writeup with code, reporting up to several-times faster generation depending on the model and workload.
Gambit Tool
An inference algorithm that prunes unpromising reasoning trajectories and immediately branches new ones from strong prefixes, keeping the hardware busy. Its authors report up to 68.5 percent fewer total tokens than standard parallel sampling with higher accuracy; code is public.
Galahad verified-reuse testbench Tool
Public testbench for the frozen-12B verified procedure cache, where a solved and independently verified problem family is answered on later instances at zero generation tokens, bit-exact. Worth poking at to understand what the claim does and does not cover -- the engine source, configuration and raw artifacts are withheld, so this demo plus the bench repo is the only inspectable surface.
GLM-5.2 on Baseten Tool
The top trending open-weight model served as a fast hosted endpoint, reported at 280+ tokens/sec on Blackwell-class hardware -- an open model you can call like a closed one.
FlashKDA Tool
Moonshot AI's MIT-licensed kernel implementation of Kimi Delta Attention, the linear-attention mechanism underneath Kimi K3, published ahead of the model weights themselves. Useful today for anyone building or serving bounded-state attention rather than a growing key-value cache.
FastFlowLM Tool
An NPU-first, GPU-free inference runtime built exclusively for AMD Ryzen AI (XDNA) NPUs, targeting long-context local LLMs at low power on laptop-class hardware; the team just joined AMD, with open install guides for Ubuntu, Arch, and more.
Doubleword (async + batch inference) Tool
Run the same models you already use, but on async and batch tiers that trade latency for a large cost cut on workloads that don't need an instant reply: long-running agents, evaluations, and bulk jobs.
DeepSeek DSpark Tool
Open-source speculative-decoding implementation using parallel tree drafting to speed up text generation with no change to the model's output - the project that topped Hacker News this week. Drop-in inference speedups for self-hosted models.
Dahl Inference Tool
Third-party inference router reselling top open-weight models (Kimi K2.6, MiniMax M2.7, GLM 5.2) at low per-token prices, currently running a 100M-free-token promotion.
DFlash 2 (Qwen3.8-27B drafter) Tool
A drop-in block-diffusion drafter for speculative decoding on Qwen3.8-27B, with documented launch commands for SGLang and vLLM. Output is provably identical to the target model; throughput gains reach 3.4x on single requests and shrink under heavy concurrency.
Cerebras Inference (Qwen 3.8 27B) Tool
Serves the open Qwen 3.8 27B at roughly 1,500 output tokens per second, with a free tier at 64k context and paid at 128k. Automatic prompt caching cuts time-to-first-token. Read the rate limits first -- the free tier's 90,000 tokens per minute lands almost exactly at the model's own output rate.