distillation
A stronger model built a wrapper that nearly doubled a weaker one's score News
Researchers had a strong model design inference-time scaffolding for weaker models, lifting their average score on four reasoning benchmarks from 0.49 to 0.91 without changing a single parameter.
A model improved itself by training only where it disagreed with itself News
U-OPSD drops the teacher entirely: it samples several attempts, takes a majority vote as a stand-in answer, then trains only on the attempts that disagreed with that vote, matching or beating methods that use real ground-truth labels.
DistillAlign explains why fast video models get prettier and more repetitive at the same time News
A new preprint shows that the standard technique for speeding up video generators optimises a mathematical objective that deliberately concentrates on safe, common outputs, and that adding a diversity-preserving loss back during that stage more than doubled the paper's coverage measure without giving up the quality gain.
Beijing says U.S. firms distilled Chinese models, and names none of them News
China's Ministry of Commerce said in a written statement on 27 July that many U.S. AI companies had distilled Chinese models during research and training, identifying no company, no model and no evidence, mirroring a U.S. accusation five days earlier that named two companies but published no logs either.
A new distillation method lets the teacher model interrupt the student mid-thought News
Researchers found that when a student model starts reasoning down a wrong path, its teacher's next word tends to be a redirection like But or Wait, and turned that disagreement into an automatic trigger for the teacher to briefly take over.
Forward and Reverse KL Divergence Lesson
KL divergence measures how far one probability distribution is from another, and it is asymmetric: swapping which distribution goes first changes whether your model tries to cover everything the target does or commit to one part of it.
The open-weights industry letter grew from 25 names to 35 - and OpenAI is on it News
A cross-industry statement titled Open Weights and American AI Leadership now lists 35 signatories on its live Microsoft-hosted page, including OpenAI, Nous Research, GitHub and Cisco, contradicting the widely shared claim that OpenAI declined to sign.
Diffusion Distillation Lesson
Diffusion distillation trains a fast student model to reproduce in a handful of steps what a slow diffusion model does in dozens, cutting image and video generation from many neural-network passes down to one to four, which is what makes real-time generative rendering possible.
US floats sanctions over AI 'distillation' as Anthropic details 16 million scraped chats News
The Treasury secretary suggested the US could sanction Chinese AI labs over model 'theft' while Anthropic and OpenAI allege large-scale unauthorized scraping of their models' outputs, but the verified record shows provider allegations and a proposed sanctions bill, not enacted policy or any proof that model weights were copied.
Model extraction attacks: stealing an AI through its own API Lesson
A model extraction attack tries to copy a machine-learning model you can only query, not download, by sending it many inputs and learning from its outputs. Depending on the goal, an attacker can clone the model's behavior, recover pieces of its internals, or reconstruct a rival model cheaply, which is exactly the fear driving today's AI 'distillation' disputes.
Grafting a Verified Solution Cache Lets a Frozen Model Skip Fine-Tuning Entirely News
A new paper shows a frozen small language model's accuracy on a hard math test can jump from 80.0% to 93.3% simply by grafting a byte-exact cache of verified solutions into it, with no weight changes at all.
Weak-to-Strong Generalization: how a worse teacher can train a better student Lesson
Weak-to-strong generalization is the finding that a strong model trained on a weaker model's flawed labels can substantially outperform its teacher, which is the only reason humans have any hope of supervising systems smarter than themselves.
A weaker model just taught a stronger one, by passing on the lesson instead of the answers News
Researchers from Tsinghua and ByteDance show that a small model's reinforcement learning gains can be distilled into a larger, already-stronger model by transferring the change in the teacher's policy rather than its outputs, letting a 1.5B teacher improve a 7B student.
Nadella calls it 'ironic' that AI labs train on the world but restrict everyone else from learning back News
Microsoft CEO Satya Nadella published an essay arguing that in the AI era buyers pay twice -- once in money, once in the proprietary knowledge they must reveal to use a model -- and called it ironic that labs claim fair-use training rights while restricting distillation.
A 4B model on your device nearly matched a 72B one - by copying its memories News
Researchers distilled a large AI agent's memory skills onto a compact 4-billion-parameter model, raising its success rate on a household-task benchmark from 4% to 78% while running three times faster than its 72B teacher.
A 32B model reaches frontier level by learning what to remember News
New research reframes agent memory as a trainable skill rather than a growing transcript - and shows that optimizing memory alone lets a 32B open model rival Claude Opus, and a 4B model leap from 4% to 78% on a benchmark.
Samsung's Trick Makes a Tiny 4B Agent Nearly Match a Model 18 Times Bigger News
Samsung R&D UK and Queen Mary University of London published DuoMem, a distillation method that took a 4-billion-parameter agent from a 4.3 percent task-success rate to 77.9 percent, nearly matching a 72-billion-parameter teacher model's 87.1 percent.
Two new papers push 'on-policy distillation' to fix privileged teachers and merge specialist skills News
DOPD and MOPD advance on-policy distillation -- training a student on its own outputs -- with DOPD routing supervision to avoid a 'privilege illusion' and MOPD merging multiple specialist RL teachers into one model without cross-domain interference.
On-Policy vs Off-Policy Learning Lesson
On-policy learning trains a model on data generated by its own current behavior, while off-policy learning trains it on data generated by something else -- an old version, a different policy, or a fixed dataset -- and the choice shapes how stable, sample-efficient, and reliable the training is.
A wave of new methods trains AI without a human answer key News
Several research groups landed on the same idea at once - improve a model by learning from its own attempts instead of expensive human labels - and the field is debating whether it really removes the labeling burden or just hides it.
NVIDIA shrinks video generation down to real time News
A new NVIDIA recipe distills slow video-generating AI into a fast version that can stream frames live and react to your actions.
Distillation: how a small AI learns from a big one Lesson
Distillation trains a smaller, cheaper model to imitate a larger, smarter one, the idea behind both efficient deployment and the 'copying' accusations now driving AI geopolitics.
Anthropic says Alibaba ran the biggest 'copy Claude' campaign yet News
Anthropic told U.S. senators that Alibaba's Qwen team quietly milked Claude for its best skills. Alibaba says nothing back, and the whole fight may be as much about price as theft.