News · 2026-07-31

Training on the best of K guesses is a third scaling axis alongside parameters and data

Researchers at the University of Illinois and Harvard have proposed a third way to scale a generative model: not more parameters and not more data, but more exploration during training. Their method generates K candidate outputs for each training example, keeps only the one closest to the real data, and trains on that. In their experiments it improved sample efficiency 6.2-fold and floating-point efficiency 4.1-fold, and - unusually - the benefit grew rather than shrank as models and datasets got larger.

Key facts

• Exploration improves sample efficiency by 6.2 times and compute efficiency by 4.1 times in the authors' experiments.
• Gains climb with scale, from 7% to 36% as data grows and from 13% to 23% as models grow.
• Posted July 29, 2026, by Alexi Gladstone, Heng Ji and Yilun Du.
• Primary source: Explorative Modeling: Unlocking a Third Pretraining Axis and End-to-End Generation, with a project page and code.

Start with the problem the method attacks. Generative models are trained to reproduce data, and the standard way to do that is to score a single prediction against a single target. That works when there is one right answer. It fails badly when there are many, because the loss quietly rewards splitting the difference. Ask a model to predict the next frame of a video where a ball could bounce left or right, and the mathematically safest prediction is a smeared ball in the middle - an output that is close to both possibilities and identical to neither. The authors frame this as the field's central awkwardness: generative modelling "is about handling distributions with many modes," and the usual fix is to factor the generation into stages, which prevents training the whole thing end to end.

Explorative Modeling factors the training loop instead of the generation. For each example the model makes K attempts; the attempt that lands nearest the real data gets the gradient; the rest are dropped. Over many steps, different attempts specialise on different possibilities rather than all drifting toward the mean. The analogy is a class of students each handing in a draft, with the teacher grading only the best one and letting the others explore: nobody is punished for taking the unpopular interpretation, so someone eventually gets it right.

The reported numbers are specific. On image generation the method lifted an already strong recipe to 1.43 FID on ImageNet without guidance - FID being a score where lower means the generated images statistically resemble real ones more closely, and 1.43 being near the best published. More interesting than any single figure is the trend: the authors report that gains from exploration "increase with scale, climbing from 7% to 36% as data scales and from 13% to 23% as models grow." Most efficiency tricks do the reverse and fade as you scale up.

The second half of the paper is a different claim: that these models can generate end to end, matching diffusion on control tasks with 16 to 256 times fewer inference steps. Diffusion models normally denoise in many small passes; here the model produces its output in far fewer.

That second result is also where the story got distorted. Community summaries compressed it into "250x faster GenAI generation," which the paper does not say. The 16-to-256x figure applies to control tasks - maze navigation and robot imitation - not to general image or text generation, and the headline image results are about how fast the model trains, not how fast it runs. It is a real result wearing someone else's hat.

Two further caveats belong in any honest reading. Exploration is not free: generating K candidates per example spends more compute during training, so the efficiency gains are measured against that cost, not on top of it. And the authors say ordinary autoregressive language models - the kind behind chat assistants - were the hardest case, because they lack a natural latent variable to explore over. Reproducibility is also incomplete: the public repository notes that code behind the headline image result and the control-task experiments is still to come. The idea is clean and the trend is striking; independent replication has not happened yet.

Primary source, verified: read the paper → (arXiv 2607.27372)

Ground Truth. (2026, July 31). Training on the best of K guesses is a third scaling axis alongside parameters and data. Ground Truth. https://groundtruth.day/news/exploring-k-guesses-per-example-is-a-third-way-to-scale-a-generative-model.html

@misc{groundtruth:exploring-k-guesses-per-example-is-a-third-way-to-scale-a-generative-model,
  title  = {Training on the best of K guesses is a third scaling axis alongside parameters and data},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/exploring-k-guesses-per-example-is-a-third-way-to-scale-a-generative-model.html}
}

Topics: generative-models · training · scaling-laws · diffusion · research