inference
OpenAI put its most intelligent model on Cerebras chips at 750 tokens a second News
OpenAI is previewing Ultrafast, a service tier that runs GPT-5.6 Sol on Cerebras hardware at up to 14 times the speed of standard processing and up to 750 output tokens per second.
DeepSeek starts charging rush-hour prices on August 17 News
DeepSeek is replacing flat API pricing with peak and off-peak rates on August 17, and the steepest change hits cached input on its Pro model, which goes up twelvefold during Beijing business hours.
A stronger model built a wrapper that nearly doubled a weaker one's score News
Researchers had a strong model design inference-time scaffolding for weaker models, lifting their average score on four reasoning benchmarks from 0.49 to 0.91 without changing a single parameter.
DeepSeek's new open model is 1.6 trillion parameters and runs 49 billion of them per token News
DeepSeek published DeepSeek-V4-Pro on Hugging Face with 1.6 trillion total parameters, 49 billion activated per token, and a one-million-token context window, making it the largest openly downloadable model of the current frontier wave.
NVIDIA built a 30B model for the boring half of agent work News
NVIDIA released Nemotron 3.5 Lightning, a 30-billion-parameter mixture-of-experts model that activates only 3 billion parameters per token and is aimed at the tool calls and validation steps that consume most of a long-running agent's time.
The Full 2.8-Trillion-Parameter Kimi K3 Now Runs on Sixteen Desktop Boxes News
An operator has the complete Kimi K3 checkpoint running across sixteen GB10 mini-workstations wired through a single 400G switch, producing roughly 21 to 25 tokens per second for one user, on hardware with a verifiable floor around $57,200.
High Bandwidth Flash Became a Spec Today, Not a Product You Can Buy News
SK hynix and Sandisk published the first standard for High Bandwidth Flash at FMS 2026, defining a NAND memory tier of up to 512GB per stack with a top bandwidth grade near three terabytes a second - with no named accelerator, price or availability date.
A llama.cpp Patch Learns Which Experts to Keep in VRAM While You Type News
An open pull request to llama.cpp tracks which mixture-of-experts submodels get used most during inference and promotes them to GPU memory on the fly, roughly doubling decode speed on an 8GB card in the author's own tests - while slowing other models down.
The Cheap 284B Rig Is Really 768GB of Server Memory News
A builder running DeepSeek V4-Flash at 33 tokens a second on two RTX 3090s is holding about 6.6GB of weights per card and roughly 170GB per instance in system memory on a four-socket enterprise server, which is where the model actually lives.
llama.cpp shipped DSpark for DeepSeek V4 Flash, and almost everyone called it the wrong name News
llama.cpp release b10228 merged speculative-decoding support for DeepSeek V4 Flash, but the new 0731 checkpoint embeds DSpark and ships no MTP at all, so the widely repeated "MTP support landed" advice points users at a head their model does not contain.
Quantizing V4 Flash's KV cache in llama.cpp changes which tokens it picks News
A community experiment found that llama.cpp's generic 8-bit KV cache leaves DeepSeek V4 Flash's average perplexity almost unchanged while altering which tokens make the model's shortlist about one time in eight, because the model's attention makes discrete block-retrieval decisions that a small numerical error can flip.
Offloading and streaming: running a model bigger than your memory Lesson
Offloading and streaming let a machine run a model far larger than its memory by keeping only the parts needed right now in fast memory and fetching the rest from system RAM or disk on demand, trading speed for capacity.
Kimi K3 runs in 8 gigabytes of RAM, at 33 seconds per token News
A hand-written C engine generates text with Moonshot's 2.8-trillion-parameter Kimi K3 using a peak of 8.24 gigabytes of RAM and no GPU, by reading the model's four-bit experts directly off disk, at a rate of roughly one token every 33 seconds.
DeepSeek's low effort setting writes more than its high setting, because the dial is just a prompt News
A reproduction across both a local copy and DeepSeek's hosted API found V4 Flash consuming substantially more tokens on its low reasoning-effort setting than on high, and the released encoder explains why: low injects no instruction at all while high prepends a paragraph demanding exhaustive deliberation.
A 284-billion-parameter model with a 3-gigabyte working set, and a 96-gigabyte disk bill News
An open-source engine called Mference runs DeepSeek V4 Flash on a 24 GB Mac with an effective memory footprint of about 3 gigabytes by streaming each token's experts off the SSD, but the checkpoint still occupies 90 to 98 gigabytes of disk and the test ran at a 4,000-token context.
OpenAI cut its cheapest model's price 80%, and credits one of its own models for making it possible News
OpenAI dropped GPT-5.6 Luna's API price by 80% and Terra's by 20% effective July 30, and says its Sol model autonomously rewrote production kernels that cut the cost of serving the model by 20%.
OpenAI says GPT-5.6 Sol autonomously rewrote the code that serves it, cutting serving costs 20% News
OpenAI published an engineering account on July 29 saying GPT-5.6 Sol, working through Codex, autonomously rewrote its production GPU kernels and redesigned its own draft model, contributing to a 20% cut in end-to-end serving cost and a 15% gain in token-generation efficiency.
NVIDIA shipped a drop-in kernel that nearly halves video generation time News
NVIDIA released code on July 28 for Sol-Attn, an attention kernel that decides which parts of a long video to compute exactly while approximating the rest inside a single pass, reporting up to 2.1 times faster video generation with no retraining and no weight changes.
A frozen 12B model answers already-solved problems at zero generation tokens News
A technical report describes a 12-billion-parameter model whose weights never change but which answers new instances of nine previously solved problem families with no generated tokens at all, scoring 180 out of 180 by executing verified stored procedures instead of reasoning again.
Chinese open models passed US models in OpenRouter token share News
OpenRouter's own usage data shows Chinese models overtaking US models in token volume in early June, with DeepSeek roughly doubling its share to 18% - driven by token-hungry agent workloads routing to the cheapest capable endpoint.
Kimi K3 is downloadable, but the floor to run it is eight datacenter GPUs News
Kimi K3's 1.56-terabyte checkpoint needs a single eight-GPU B300 or MI355X node as its practical minimum, and no version of llama.cpp can load it today, so open weights currently mean operator-scale rather than local.
Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math Lesson
Generating text with a language model is limited by how fast weights can be moved from memory into the processor, not by how fast the processor can multiply, which is why most of a GPU sits idle during inference.
Model Routing and Cascades: Spending Frontier Money Only Where It Buys Something Lesson
Model routing sends each request to the cheapest model that can handle it, and cascades escalate to a stronger model only when a cheaper one is not confident, which cuts serving costs substantially because most production traffic is easy.
AMD and Cerebras split AI inference across two different chips News
AMD and Cerebras announced a joint inference offering on July 23 in which AMD's Helios racks process the prompt and Cerebras's wafer-scale engine generates the tokens, claiming up to five times the tokens per watt of a Cerebras-only setup.
GigaToken Claims a 1,000x Tokenizer Speedup, But the Real Serving Win Is Single-Digit News
A new tokenizer called GigaToken reports roughly 1,000 times the offline throughput of Hugging Face Tokenizers, but its benchmark is not apples-to-apples and its real inference payoff is a single-digit latency improvement.
Google's two opposite bets: a Gemini-specialized chip and an EU order to open Android AI News
Google is reportedly designing a server chip called Frozen v2 that hardwires Gemini's architecture for six-to-ten times more tokens per watt, even as the European Commission adopted binding measures forcing Android to open eleven AI capabilities to rival assistants, making Google simultaneously bet on locking Gemini into silicon and being forced to unlock Gemini's Android advantages.
Gemini 3.6 Flash: Google ships a faster worker, not a bigger brain News
Google released Gemini 3.6 Flash into general availability, and independent benchmarks show it streams output nearly twice as fast as 3.5 Flash and costs less per task while scoring the same on a leading intelligence index, though it still takes a conspicuous 11-plus seconds to start responding.
Google Falls Off One Leaderboard's Top 15, as a Report Describes a Gemini-Specific Chip News
Google dropped out of the top 15 on LLM Stats' composite leaderboard while remaining its fastest model, and Reuters separately reported an unannounced Gemini-specific inference chip.
AMD Absorbs FastFlowLM Team to Build GPU-Free NPU Inference News
AMD announced on July 17, 2026 that the FastFlowLM team has joined its Artificial Intelligence Group to build out an NPU-first, GPU-free local inference stack for Ryzen AI laptops.
Prompt Caching: Why AI Agents Pay Once to Read, Then Read for Pennies Lesson
Prompt caching lets an AI provider store the processed form of a repeated chunk of text -- like a long system prompt -- so it can be reused across requests at a fraction of the cost, instead of being re-processed every time.
DeepSeek is designing its own AI chip -- and raising outside money for the first time News
Chinese AI startup DeepSeek is developing its own chip aimed at running trained models rather than training them, and is simultaneously raising its first-ever outside capital -- about $7 billion at a $52-59 billion valuation.
New methods page an AI's memory in and out like an operating system News
Two new techniques treat a language model's long-context memory like an operating system's memory hierarchy - keeping coarse summaries on the GPU and paging compressed detail out to the CPU - with one, SeKV, cutting GPU memory use by 53% at 128,000 tokens while avoiding the errors that plague simpler methods.
A startup router is giving away 100 million tokens of Kimi, MiniMax and GLM News
API aggregator Dahl Inference is handing out 100 million free tokens across top open-weight Chinese models like Kimi K2.6 and MiniMax M2.7 - not a price cut from the labs themselves, but a router burning money to win users amid a glut of cheap compute.
A $4-per-million open model is coming for the frontier's 90% margin News
GLM-5.2, an open-weights model priced at under a fifth of Opus, scores as the top open model and 4th overall - and a widely-shared essay argues it is the first real threat to frontier labs' ~90% inference margins.
Program-as-Weights compiles a plain-English spec into a tiny model you run on a laptop News
A new method called Program-as-Weights uses a 4-billion-parameter 'compiler' to turn a natural-language task description into a small weight file that a frozen 0.6B model runs, matching a 32B model's quality while using about one-fiftieth the memory and running at 30 tokens a second on a MacBook.
Test-Time Compute: Spending More Thinking at the Moment You Ask Lesson
Test-time compute is the idea of making an AI smarter not by retraining it, but by letting it do more work the moment you ask a question: think longer, or generate many answers and pick one.
GPT-5.5 Codex Keeps Cutting Its Own Reasoning Off at Exactly 516 Tokens News
A GitHub analysis of 390,195 coding-session responses found GPT-5.5 disproportionately cuts off its own reasoning at exactly 516 tokens, a pattern likely caused by a batching bug rather than an intentional change.
Temperature and top-p: how an AI actually picks its next word Lesson
A language model does not know its next word - it produces a list of odds and then rolls dice. The rules of that dice roll are why the same prompt gives you a boring answer one day and a wild one the next.
Ollama nearly doubles Gemma's speed on Macs by guessing ahead News
A free local-AI tool now runs Google's Gemma model far faster on Apple computers using a trick where a small model drafts words and the big one checks them in bulk.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
The trick that makes AI type faster just hit the top of Hacker News News
A small model guesses ahead and a big model checks the work in parallel - and this week two efforts pushing that idea, DeepSeek's DSpark and JetSpec, lit up the front page while the community argued over whether it's truly 'lossless.'
Speculative Decoding: How AI Types Faster Without Changing a Word Lesson
A small, fast model guesses the next few words and a big, slow model checks them all in one pass - producing the exact same output, just quicker. The trick behind a lot of modern AI speedups.
Quantization: Shrinking AI Models to Run on Modest Hardware Lesson
Storing a model's numbers with less precision - 8, 4, or even fewer bits instead of 16 - makes it dramatically smaller and faster, often with almost no loss in quality. It's why big models can run on a laptop or a single GPU.
Frontier AI is getting more expensive while open models keep getting cheaper News
Closed frontier models are raising prices and tightening access just as Chinese open-weight models slash theirs, a structural reversal with big consequences for who builds with AI.
Chain-of-thought: why making an AI think out loud makes it smarter Lesson
Asking a model to work through a problem step by step, instead of blurting an answer, dramatically improves it on hard tasks. Here is why that simple trick works, what it really buys the model, and where it backfires.
Training vs inference: the two very different jobs inside every AI Lesson
Why building an AI model and using it are separate worlds with separate costs, and why that split explains custom chips, model prices, and where the real money in AI actually goes.
OpenAI designs its own chip to run its models News
With Broadcom, OpenAI unveiled a custom chip built for one job: serving its AI models cheaply.
Two labs race to make AI write whole paragraphs at once instead of word by word News
Diffusion text models generate in parallel blocks rather than left to right; Google's open DiffusionGemma and Inception's Mercury 2 are now in a head-to-head over speed.
Suddenly, downloadable AI models look like an insurance policy News
With a top hosted model pulled overnight, a flood of powerful open models you can run yourself -- and run fast -- is being reframed from hobby to risk management.
vLLM v0.23.0 Tool
The widely-used open engine for serving language models fast and cheaply. The latest release adds smarter memory handling for long conversations and faster GPU execution.
vLLM recipe for DeepSeek V4 Flash Tool
An official vLLM recipe page with working launch commands for serving V4 Flash across several hardware configurations, including the flag that turns on the DSpark speculative-decoding module and the FP8 KV-cache and expert-parallel settings DeepSeek recommends.
vLLM DeepSeek-V4 support Tool
vLLM shipped serving support for DeepSeek-V4's compressed long-context attention, including hybrid KV-cache management, multiple cache page sizes, kernel fusion and multi-stream partitioning. The engineering post documents the recipe and the hardware it assumes.
vLLM (Kimi K3 support) Tool
The widely used open-source serving engine landed day-zero Kimi K3 support with a documented recipe, an FAQ on minimum hardware, and a K3-specific DSpark draft model for speculative decoding that roughly triples single-user throughput.
llama.cpp b10228 Tool
The release that adds DeepSeek V4 Flash's embedded DSpark speculative-decoding head, plus a converter that can split the draft tensors into a separate GGUF. Gains are workload-dependent: roughly 2x decode on large multi-GPU setups, and a measured slowdown on a 24 GB card with CPU offload.
llama.cpp Tool
The lean, fast engine that makes big models run on ordinary laptops; powers much of the local-AI ecosystem.
bitnet.cpp Tool
Microsoft's official inference framework for 1.58-bit ternary language models, built on llama.cpp with optimized CPU and GPU kernels for running very heavily compressed models on ordinary hardware.
SwiftLM Tool
An MLX-based runtime for Apple silicon whose --stream-experts mode reads mixture-of-experts weights straight off an NVMe SSD, letting a machine run models several times larger than its RAM.
Sol-Attn (Sol-Engine) Tool
NVIDIA's drop-in sparse attention kernel for long-video diffusion transformers, released July 28 for HunyuanVideo-13B and Wan2.1-T2V-14B. Screens compressed key/value blocks inside a single online-softmax pass, so exact attention goes where it matters and skipped blocks get an approximate correction. Training-free, no weight changes, reported up to 2.1x for generation and 2.3x for editing. The repo marks end-to-end re-benchmarks for the two integrated pipelines as pending.
SGLang v0.5.13 Tool
A high-performance open serving engine for language models. The new version turns on faster 'guess-ahead' decoding by default and trims scheduling overhead for quicker responses.
SGLang (Kimi K3 cookbook) Tool
Alternative open-source serving engine with day-zero K3 support and a step-by-step deployment cookbook. Its writeup documents how prefix caching, paging and prefill/decode disaggregation were rebuilt to handle K3's mix of recurrent and key-value state.
SGLang Tool
The other serving stack DeepSeek's official 0731 model card documents as supporting DSpark directly, alongside the recommended FP8 key-value cache and FP4 indexer cache configuration for V4 Flash.
Program-as-Weights Tool
Turns a plain-English task spec into a small weight file that a frozen 0.6B model runs locally -- matching a 32B model's quality at roughly one-fiftieth the memory and about 30 tokens/sec on a MacBook M3. Open repo and site for compiling cheap, offline 'fuzzy' text programs.
Ollama 0.31 Tool
Run open models on your own computer; the new version nearly doubles Gemma's speed on Apple Silicon using multi-token prediction, on by default.
NVIDIA Nemotron 3.5 Lightning Tool
A 30B mixture-of-experts model with only 3B parameters active per token, trained for the high-volume half of agent work: tool calls, result validation and subagent delegation. NVFP4 and BF16 checkpoints, with weights, training data and recipes released under a permissive licence.
NInfer Tool
A focused inference engine that runs Qwen 3.6 models on one RTX 5090 with a 262,000-token context using an INT8 key-value cache, reporting roughly 188 tokens per second at 250,000 tokens of context. Methodology, seeds and limits are published openly.
Modular MAX + Mojo Tool
A programming language (Mojo) and compiler/runtime (MAX) for running AI models efficiently across different hardware instead of being locked to one chip vendor; now being acquired by Qualcomm but still openly available to developers.
Mference Tool
Runs DeepSeek V4 Flash on Apple silicon by keeping the shared core, attention and cache resident while streaming each token's routed experts off the SSD. Publishes an unusually honest memory budget: about 3 GB working set, 90 to 98 GB on disk, tested at a 4,000-token context on a 24 GB Mac, with no quality parity test yet.
LvLLM Tool
A community inference runtime specialised in hybrid CPU and GPU execution of mixture-of-experts models, with NUMA-aware scheduling, expert weight management, and MXFP4 quantization kernels. Its DeepSeek V4 build publishes a working dual-RTX-3090 launch configuration at 22K context.
Kimi-K3-DSpark Tool
Inferact's draft model for Kimi K3. It proposes seven tokens at a time for K3 to verify and accept or discard, and its block-diffusion backbone shares K3's attention-cache layout so no second cache format is needed. This is the component behind the 21-25 tokens per second measured on a sixteen-node GB10 cluster running the full K3 checkpoint.
JetSpec Tool
Parallel tree-drafting speculative decoding aiming for large, lossless inference speedups; project page and writeup with code, reporting up to several-times faster generation depending on the model and workload.
Galahad verified-reuse testbench Tool
Public testbench for the frozen-12B verified procedure cache, where a solved and independently verified problem family is answered on later instances at zero generation tokens, bit-exact. Worth poking at to understand what the claim does and does not cover -- the engine source, configuration and raw artifacts are withheld, so this demo plus the bench repo is the only inspectable surface.
GLM-5.2 on Baseten Tool
The top trending open-weight model served as a fast hosted endpoint, reported at 280+ tokens/sec on Blackwell-class hardware -- an open model you can call like a closed one.
FlashKDA Tool
Moonshot AI's MIT-licensed kernel implementation of Kimi Delta Attention, the linear-attention mechanism underneath Kimi K3, published ahead of the model weights themselves. Useful today for anyone building or serving bounded-state attention rather than a growing key-value cache.
FastFlowLM Tool
An NPU-first, GPU-free inference runtime built exclusively for AMD Ryzen AI (XDNA) NPUs, targeting long-context local LLMs at low power on laptop-class hardware; the team just joined AMD, with open install guides for Ubuntu, Arch, and more.
Doubleword (async + batch inference) Tool
Run the same models you already use, but on async and batch tiers that trade latency for a large cost cut on workloads that don't need an instant reply: long-running agents, evaluations, and bulk jobs.
DeepSeek DSpark Tool
Open-source speculative-decoding implementation using parallel tree drafting to speed up text generation with no change to the model's output - the project that topped Hacker News this week. Drop-in inference speedups for self-hosted models.
Dahl Inference Tool
Third-party inference router reselling top open-weight models (Kimi K2.6, MiniMax M2.7, GLM 5.2) at low per-token prices, currently running a 100M-free-token promotion.