efficiency
A 150M model set an ARC-AGI record for cost, not score News
Pathway's BDH-CQ solves reasoning puzzles by iterating in a latent workspace instead of writing out its thinking, reaching 29.5% on public ARC-AGI-1 at seven hundredths of a cent per task and setting a record in cost efficiency rather than accuracy.
Two-bit models now run on every major llama.cpp backend News
The official 2-bit weight format merged into llama.cpp now works on mainline CPU, Metal, Vulkan, and CUDA after the CUDA path landed upstream, completing a backend sweep that started as an ARM-only contribution.
An agent's memory records what you said; this compiler records what you did News
A single-author paper argues computer-use agents waste frontier inference re-deriving routines their user already performed, and presents a model-free compiler that turns raw screen capture into a context block 86 times smaller that an agent reads with 98.4 percent accuracy.
Ant published Ling-3.0-flash's weights under plain MIT, with no rider attached News
Ant Group's InclusionAI lab published the full weights for its 124-billion-parameter Ling-3.0-flash model on Hugging Face this week under an unmodified MIT license, with no acceptable-use policy, revenue threshold, or branding requirement anywhere in the release.
Model merging: combining two fine-tuned models by averaging their weights Lesson
Model merging combines two or more models that share a common ancestor by arithmetic on their weights - averaging them, adding their differences, or interpolating between them - producing a single model with both sets of skills and no additional training.
OpenAI cut its cheapest model's price 80%, and credits one of its own models for making it possible News
OpenAI dropped GPT-5.6 Luna's API price by 80% and Terra's by 20% effective July 30, and says its Sol model autonomously rewrote production kernels that cut the cost of serving the model by 20%.
AI search agents get better when relevance tells them where to look, not what to read News
Researchers at Tencent rebuilt relevance as a guide for how a search agent traverses a corpus rather than as a ranked list of documents, cutting the agent's tool calls by roughly a sixth while raising accuracy.
A robot control model now runs 32 times a second on a gaming GPU, in under a gigabyte News
TurboVLA reaches real-time robot control at 32 Hz using 0.9GB of memory on a consumer RTX 4090, by removing the large language model from the control loop entirely rather than compressing it.
OpenAI says GPT-5.6 Sol autonomously rewrote the code that serves it, cutting serving costs 20% News
OpenAI published an engineering account on July 29 saying GPT-5.6 Sol, working through Codex, autonomously rewrote its production GPU kernels and redesigned its own draft model, contributing to a 20% cut in end-to-end serving cost and a 15% gain in token-generation efficiency.
NVIDIA shipped a drop-in kernel that nearly halves video generation time News
NVIDIA released code on July 28 for Sol-Attn, an attention kernel that decides which parts of a long video to compute exactly while approximating the rest inside a single pass, reporting up to 2.1 times faster video generation with no retraining and no weight changes.
A frozen 12B model answers already-solved problems at zero generation tokens News
A technical report describes a 12-billion-parameter model whose weights never change but which answers new instances of nine previously solved problem families with no generated tokens at all, scoring 180 out of 180 by executing verified stored procedures instead of reasoning again.
Microsoft lets the video codec pick which pixels the model sees News
Microsoft's Mage-VL reuses a video file's own compression decisions to choose which image patches a vision model processes, cutting visual tokens by over 75% and reporting up to a 3.5x speedup over uniform frame sampling.
Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math Lesson
Generating text with a language model is limited by how fast weights can be moved from memory into the processor, not by how fast the processor can multiply, which is why most of a GPU sits idle during inference.
Baidu's OCR Model Reads Dozens of Pages in One Pass Without Its Memory Growing News
Unlimited OCR replaces the decoder's attention layers with a design that holds the key-value cache at a constant size, so transcription speed does not degrade as the document gets longer.
A complete text-to-speech system now fits in 9.4 million parameters News
Inflect-Micro-v2 packs an entire English speech synthesis stack, including the waveform decoder, into 9,356,513 parameters that run locally with no external vocoder or API.
Three separate tricks dropped the hardware floor for local AI in one day News
A 26-billion-parameter Gemma model ran on an iPhone by streaming expert weights from storage, AMD shipped a sparse mixture-of-experts model activating 2.8 billion parameters per token, and a llama.cpp fork began saving conversation caches to disk - three unrelated attacks on three different bottlenecks.
Mixed-precision training: why models are trained in half-broken numbers on purpose Lesson
Modern models are trained using 16-bit and even 8-bit numbers instead of the 32-bit standard, roughly doubling speed and halving memory, by carefully keeping full precision exactly where the arithmetic would otherwise fall apart.
Giving experts no memory cuts optimizer state from 50 gigabytes to 1.3 News
A single-author preprint shows that a mixture-of-experts model can drop momentum entirely for its expert layers, shrinking persistent optimizer state from 50.55 gigabytes to 1.29 with almost no effect on final quality.
Ant's Ling-3.0-flash goes live free: 124 billion parameters, 5 billion doing the work News
Ant released Ling-3.0-flash on July 23, a 124-billion-parameter model that activates only about 4% of itself per token, with a 256,000-token context and free access on OpenRouter and Vercel's AI Gateway.
AMD and Cerebras split AI inference across two different chips News
AMD and Cerebras announced a joint inference offering on July 23 in which AMD's Helios racks process the prompt and Cerebras's wafer-scale engine generates the tokens, claiming up to five times the tokens per watt of a Cerebras-only setup.
Linear Attention Lesson
Linear attention rewrites the transformer's attention math so its cost grows in a straight line with sequence length instead of exploding as the square, letting models handle very long inputs cheaply, at the price of a compressed memory of the past.
Diffusion Distillation Lesson
Diffusion distillation trains a fast student model to reproduce in a handful of steps what a slow diffusion model does in dozens, cutting image and video generation from many neural-network passes down to one to four, which is what makes real-time generative rendering possible.
Nanbeige4.2-3B reuses one 22-layer stack twice to punch above its size News
A Chinese lab released Nanbeige4.2-3B, a small open-weight model that runs its 22 transformer layers twice in sequence to get 44 layers of depth from one set of weights, posting benchmark numbers rivaling models three times its size, though the results are vendor-reported and a widely repeated 'beats 4x its size' claim does not survive clean accounting.
Looped transformers: reusing the same layers to think deeper Lesson
A looped transformer runs its input through the same stack of layers more than once, so it gets the processing depth of a deep network while storing the weights of a shallow one. It trades extra compute for a smaller memory footprint, and it is the trick behind small models that punch above their parameter count.
Gemini 3.6 Flash: Google ships a faster worker, not a bigger brain News
Google released Gemini 3.6 Flash into general availability, and independent benchmarks show it streams output nearly twice as fast as 3.5 Flash and costs less per task while scoring the same on a leading intelligence index, though it still takes a conspicuous 11-plus seconds to start responding.
VideoChat3 Halves Video-Model Latency by Compressing Space and Time First News
VideoChat3, an open 4-billion-parameter video model, compresses frames across space and time before the language model, roughly halving latency versus a comparable model.
LongStraw Makes Million-Token RL Training Possible on 8 GPUs - But Its Code Doesn't Run Yet News
A new systems paper called LongStraw shows reinforcement-learning post-training can execute on prompts beyond 2 million tokens on a fixed 8-GPU budget by scoring the shared prompt once without gradients and backpropagating only through the short generated response, though the authors call this proof of execution capacity rather than full training correctness, and the public code is not yet runnable.
Grafting a Verified Solution Cache Lets a Frozen Model Skip Fine-Tuning Entirely News
A new paper shows a frozen small language model's accuracy on a hard math test can jump from 80.0% to 93.3% simply by grafting a byte-exact cache of verified solutions into it, with no weight changes at all.
AMD Absorbs FastFlowLM Team to Build GPU-Free NPU Inference News
AMD announced on July 17, 2026 that the FastFlowLM team has joined its Artificial Intelligence Group to build out an NPU-first, GPU-free local inference stack for Ryzen AI laptops.
KronQ makes 2-bit models work where the standard method collapses News
Adding gradient information to model compression takes a 2-bit LLaMA-3-70B from complete gibberish under the dominant method to a functioning model, suggesting the standard approach has been missing half the picture since 2022.
Bonsai puts a 27B model on your phone, and shows what breaks News
PrismML shrank a 27-billion-parameter model from 54 gigabytes to 3.9 by storing each weight as a single bit, and published the benchmark table showing tool calling and vision falling off a cliff.
FlashAttention: making attention fast by respecting the memory hierarchy Lesson
FlashAttention is an exact attention algorithm that runs much faster and uses far less memory by tiling the computation to keep it in the GPU's fast on-chip memory - never writing the giant attention matrix to slow memory - which is why it became the default attention kernel in modern transformers.
Apple's on-device SpeechAnalyzer beats Whisper Small using about a third of the compute News
Apple's new on-device SpeechAnalyzer API cut errors roughly fourfold versus its legacy recognizer and outperformed OpenAI's Whisper Small while using about a third of the compute, in an English-only single-vendor benchmark.
Someone put a wiretap on two coding agents. One sends five times more before you type a word. News
An applied-research firm metered the traffic of two AI coding agents and found Claude Code sends about 33,000 tokens of fixed overhead before your prompt arrives, versus about 7,000 for OpenCode.
State Space Models and Mamba Lesson
State space models are a transformer alternative that processes a sequence by carrying a fixed-size running summary forward one step at a time, giving them linear cost with length and constant memory per token -- which is why models like Mamba can handle very long inputs cheaply.
Nvidia's 'Puzzle' pipeline shrinks a 120B model to 75B and roughly doubles serving throughput News
Nvidia's Nemotron-3-Puzzle compresses a 120-billion-parameter model down to 75 billion using a multi-stage pipeline of pruning, distillation, and quantization -- roughly doubling throughput on a B200 node and lifting single-H100 concurrency at long context from one request to eight.
A new attention method learns what to ignore, and reads 64x more text News
Tencent's Hunyuan team introduced HiLS, a sparse-attention method that learns end-to-end which parts of a long document to focus on, matching full attention while handling context 64 times longer than it was trained on.
New methods page an AI's memory in and out like an operating system News
Two new techniques treat a language model's long-context memory like an operating system's memory hierarchy - keeping coarse summaries on the GPU and paging compressed detail out to the CPU - with one, SeKV, cutting GPU memory use by 53% at 128,000 tokens while avoiding the errors that plague simpler methods.
A 4B model on your device nearly matched a 72B one - by copying its memories News
Researchers distilled a large AI agent's memory skills onto a compact 4-billion-parameter model, raising its success rate on a household-task benchmark from 4% to 78% while running three times faster than its 72B teacher.
Program-as-Weights compiles a plain-English spec into a tiny model you run on a laptop News
A new method called Program-as-Weights uses a 4-billion-parameter 'compiler' to turn a natural-language task description into a small weight file that a frozen 0.6B model runs, matching a 32B model's quality while using about one-fiftieth the memory and running at 30 tokens a second on a MacBook.
Samsung's Trick Makes a Tiny 4B Agent Nearly Match a Model 18 Times Bigger News
Samsung R&D UK and Queen Mary University of London published DuoMem, a distillation method that took a 4-billion-parameter agent from a 4.3 percent task-success rate to 77.9 percent, nearly matching a 72-billion-parameter teacher model's 87.1 percent.
This method compiles plain English into a tiny model that rivals a 32B giant News
A new paper, Program-as-Weights, uses a 4B compiler to turn natural-language specs into small neural adapters that let a 0.6B model match a 32B model on fuzzy tasks, running at about 30 tokens per second on a laptop.
A training-free trick makes AI image generation up to 10x faster News
MrFlow speeds up diffusion image models like FLUX by generating a rough image at low resolution, sharpening it with a fast GAN, and refining details at full resolution -- reaching roughly 10x end-to-end speedup with almost no quality loss and no retraining.
Sparse Attention Lesson
Sparse attention lets a transformer skip most of the pairwise comparisons between tokens, so instead of every token attending to every other token, each one attends to a chosen subset -- which is what makes million-token context windows affordable.
Fine-tuning and LoRA: teaching an old model a new job without retraining it Lesson
You almost never train an AI from scratch. You take one that already knows language and nudge it toward your specific task - and a trick called LoRA lets you do that by adding a tiny sticky note instead of rewriting the whole brain.
The KV cache: why AI gets slower and hungrier the longer it talks Lesson
The hidden notebook that lets a model avoid re-reading every previous word - and the single biggest reason long context is expensive.
The trick that makes AI type faster just hit the top of Hacker News News
A small model guesses ahead and a big model checks the work in parallel - and this week two efforts pushing that idea, DeepSeek's DSpark and JetSpec, lit up the front page while the community argued over whether it's truly 'lossless.'
Speculative Decoding: How AI Types Faster Without Changing a Word Lesson
A small, fast model guesses the next few words and a big, slow model checks them all in one pass - producing the exact same output, just quicker. The trick behind a lot of modern AI speedups.
Quantization: Shrinking AI Models to Run on Modest Hardware Lesson
Storing a model's numbers with less precision - 8, 4, or even fewer bits instead of 16 - makes it dramatically smaller and faster, often with almost no loss in quality. It's why big models can run on a laptop or a single GPU.
Distillation: how a small AI learns from a big one Lesson
Distillation trains a smaller, cheaper model to imitate a larger, smarter one, the idea behind both efficient deployment and the 'copying' accusations now driving AI geopolitics.
Mixture of Experts: The Committee Inside a Giant Model Lesson
Why the biggest AI models are not really one big brain but a large team of specialists, only a few of whom wake up for any given word -- the trick that lets a model be huge and fast at the same time.
A small but elegant idea: putting 'experts' inside the attention layer News
Grouped Query Experts brings the mixture-of-experts trick into attention, activating only half a model's query heads per token while matching the full version -- at least at small scale.
A Classic Efficiency Trick Just Moved Into a New Part of the AI News
For years, the committee-of-specialists design that keeps big models fast lived in one layer of the network. A clean new result shows it works in the attention layer too, halving some of the work for free.
Teaching AI with rewards — minus the expensive second model that grades it News
The standard way to polish a model with rewards quietly runs a second 'critic' model alongside it. A new method derives the critic's judgment from the model itself, dropping the extra cost.
Scaling laws — does bigger always mean better? Lesson
For years, AI progress ran on a simple recipe: make the model bigger, feed it more data, get a better model. That pattern is real and predictable — but it has limits and surprises. Here's what scaling laws actually say.
Robots may not need to picture the future as video to act on it News
Generating a full imagined video of what comes next is expensive. A new method skips it — pulling a robot's next move straight from the inner workings of an image-editing model.
A world model that thinks in loops instead of stacking layers News
Instead of building an ever-deeper neural network to simulate the future, a new design re-runs one small block over and over — doing comparable work with a fraction of the size.
Faster AI training by quietly cloning the model News
Teaching a model with rewards is slow because it has to write out endless practice answers. A new trick: make a cheap, shrunk-down copy of the model to crank those out faster.
Do robots even need to imagine the movie? News
The common belief is that a robot needs to imagine a video of what happens next to plan. A new method says no — imagine a single still frame, and don't even fully draw it.
A tiny image-fixer keeps up with a model fifty times its size News
Filling in the missing parts of an image usually takes a huge model. This one is a small fraction of the size and far faster, yet matches a system far bigger than it.
Voodoo Quant Tool
A per-tensor sensitivity-aware quantization method that spends more bits on important tensors, claiming large divergence reductions over standard llama.cpp and Unsloth quants, especially at 1-bit and 2-bit; GGUF files run in unmodified llama.cpp.
Sol-Attn (Sol-Engine) Tool
NVIDIA's drop-in sparse attention kernel for long-video diffusion transformers, released July 28 for HunyuanVideo-13B and Wan2.1-T2V-14B. Screens compressed key/value blocks inside a single online-softmax pass, so exact attention goes where it matters and skipped blocks get an approximate correction. Training-free, no weight changes, reported up to 2.1x for generation and 2.3x for editing. The repo marks end-to-end re-benchmarks for the two integrated pipelines as pending.
Qwen3-Next-80B-A3B-Instruct Tool
Alibaba's efficiency-focused open-weight model (80B total / 3B active, 512 experts) with 262K native context to ~1M, built around hybrid attention, high-sparsity MoE, and multi-token prediction; the model card claims roughly 10x inference throughput past 32K context versus a dense 32B baseline.
Program-as-Weights demo Tool
A public demo and code for compiling natural-language task specs into tiny neural artifacts that run locally on a frozen small model, matching much larger models on narrow fuzzy tasks.
Program-as-Weights Tool
Turns a plain-English task spec into a small weight file that a frozen 0.6B model runs locally -- matching a 32B model's quality at roughly one-fiftieth the memory and about 30 tokens/sec on a MacBook M3. Open repo and site for compiling cheap, offline 'fuzzy' text programs.
OpenCode Tool
An open coding agent shown this week to send a fraction of the fixed token overhead of some rivals, with a stable prompt-cache prefix; works against frontier and local models alike.
Nemotron-3-Puzzle-75B Tool
Nvidia's compressed 75B open model (from a 120B parent) with roughly double the serving throughput and 8x long-context concurrency on a single H100; weights on Hugging Face.
Mage-VL Tool
Microsoft's codec-native multimodal model that reuses a video file's own bit allocation to pick visual tokens, reporting over 75% fewer tokens and up to 3.5x faster inference than uniform frame sampling. Works with H.264, HEVC and DCVC-RT.
JetSpec Tool
Parallel tree-drafting speculative decoding aiming for large, lossless inference speedups; project page and writeup with code, reporting up to several-times faster generation depending on the model and workload.
Inflect-Micro-v2 Tool
A complete English speech synthesis stack in 9,356,513 parameters, waveform decoder included, producing 24 kHz mono audio locally with no external vocoder or API. One fixed synthetic male voice, no cloning, flatter prosody than large systems - but it runs anywhere.
FastFlowLM Tool
An NPU-first, GPU-free inference runtime built exclusively for AMD Ryzen AI (XDNA) NPUs, targeting long-context local LLMs at low power on laptop-class hardware; the team just joined AMD, with open install guides for Ubuntu, Arch, and more.
DeepSeek DSpark Tool
Open-source speculative-decoding implementation using parallel tree drafting to speed up text generation with no change to the model's output - the project that topped Hacker News this week. Drop-in inference speedups for self-hosted models.
Bonsai 27B (GGUF) Tool
PrismML's roughly 27.8-billion-parameter Qwen-derived model trained with 1-bit binary or 1.58-bit ternary weights end to end, which the company says fits in about 4 GB and runs on phone-class hardware. Performance figures are vendor-reported and not independently replicated.
Bonsai 27B Tool
PrismML's 1-bit and 1.58-bit builds of Qwen3.6 27B, compressing a 54 GB model to 3.9 GB (binary) or 5.9 GB (ternary) and running at roughly 11 tokens per second on an iPhone 17 Pro. The release ships an honest benchmark table showing the cost: instruction following, tool calling, and vision all degrade sharply, and the vendor states agentic coding is not a strong target of this release.