News · 2026-07-07

A giant benchmark tested 24 optimizers - and AdamW's edge held up

OmniOpt, a sweeping new benchmark, put more than two dozen modern training optimizers through an identical, controlled bake-off across model sizes from 60 million to 1 billion parameters - and its headline result is a splash of cold water. No challenger cleanly dethrones AdamW, the workhorse optimizer everyone already uses. An optimizer's edge, it turns out, is real but conditional, depending on scale, task, and how much you're willing to tune.

Key facts

• OmniOpt benchmarks 24+ optimizers (Adam-family, Muon, Shampoo, and more) under a controlled-variable protocol; code at OpenRaiser/OmniOpt.
• It sweeps four LLaMA-3 scales (60M to 1B parameters) trained on the C4 dataset, then transfers winners to FineWeb-Edu.
• Only the optimizer's hyperparameters are tuned; architecture and data are locked, isolating the optimizer's true contribution.
• Conclusion: the "no free lunch" reality holds - no single optimizer wins across all scales and budgets.

Start with why this benchmark needed to exist. The optimizer is the part of training that decides, after the model sees a batch of data and computes how wrong it was, exactly how to nudge its billions of weights. AdamW has been the default for years, and every so often a new optimizer - Muon, Shampoo, a fresh Adam variant - arrives with a paper claiming it trains faster or reaches a lower loss. The trouble is that these claims are almost never comparable. One paper tests on a tiny model, another on a huge one; one tunes its new method carefully while leaving the baseline at default settings; the datasets and architectures differ. The result is a decade of "our optimizer beats AdamW" claims that don't stack up against each other.

OmniOpt's real product is not a winner but a protocol - a fair fight. It locks everything that isn't the optimizer: same architecture, same data, same training budget. Then it tunes only the optimizer's own knobs - the learning rate, the momentum terms, the numerical-stability epsilon - so that any difference in the final model can be attributed to the optimizer and nothing else. It runs this across four model scales so you can see whether an advantage that shows up at 60 million parameters survives at a billion, and it transfers the best performers to a second dataset to check whether the ranking holds or was an artifact of the first. This is the controlled-variable discipline that individual optimizer papers rarely can afford, and it is the whole value.

Conceptually, OmniOpt also offers a unifying lens: it treats these 24 optimizers not as a zoo of unrelated tricks but as instances of a common five-stage "meta-pipeline" built around norm-constrained update rules. In that framing, the differences between Adam, Muon, and Shampoo become choices within a shared template - how to precondition the gradient, how to constrain the size of each step - rather than fundamentally different beasts. That's useful because it turns "which optimizer?" into "which knobs in the pipeline?", a more tractable question.

The deflating-but-important finding is that the honest answer is "it depends." An optimizer that pulls ahead at small scale can lose its lead as the model grows; one that shines with a lavish tuning budget may be no better than AdamW when you only get a few runs to find good hyperparameters. This is the "no free lunch" theorem showing up empirically: there is no universally best optimizer, only best-for-your-situation ones. For practitioners, that is genuinely actionable - it means the right move is usually to stick with well-tuned AdamW unless you have a specific scale and budget where a challenger has been shown to win, not to chase every new optimizer that trends.

The caveat cuts both ways. A benchmark that tops out at a billion parameters can't rule out that some optimizer's advantage only emerges at the tens-of-billions scale where frontier models actually live - and several of the more exotic optimizers were designed precisely for that regime. Training economics at scale are exactly where a few percent of efficiency translates into millions of dollars, so "AdamW is fine" at 1B doesn't automatically settle the question at 100B. What OmniOpt settles is the methodology: from now on, an optimizer claiming to beat AdamW has a fair, public, controlled arena it has to win in - and so far, the incumbent is still standing.

Primary source, verified: read the paper → (arXiv 2607.04033)

Ground Truth. (2026, July 7). A giant benchmark tested 24 optimizers - and AdamW's edge held up. Ground Truth. https://groundtruth.day/news/a-cookbook-benchmarks-24-optimizers-against-adamw.html

@misc{groundtruth:a-cookbook-benchmarks-24-optimizers-against-adamw,
  title  = {A giant benchmark tested 24 optimizers - and AdamW's edge held up},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/news/a-cookbook-benchmarks-24-optimizers-against-adamw.html}
}

Topics: optimizers · training · adamw · benchmark · research