Learn · Intermediate

Optimizers: how Adam and AdamW turn gradients into learning

An optimizer is the rule a neural network uses to decide how to change its weights after each mistake. If gradient descent tells you which direction is downhill, the optimizer decides how big a step to take, in what refined direction, and how much to trust the momentum you've built up. Adam and its successor AdamW became the near-universal default for training modern AI because they adapt the step size for every single weight, making training dramatically faster and less finicky than the plain version.

From one learning rate to millions

Start with what plain gradient descent actually does. After the network sees a batch of data, backpropagation computes a gradient - a number for each weight saying "nudging you this way would reduce the error." Vanilla gradient descent then updates every weight by the same fixed fraction of its gradient, called the learning rate. That fraction is a single global knob, and it is a terrible one-size-fits-all. Some weights sit on steep parts of the error landscape and need tiny steps or they overshoot; others sit on flat plateaus and need big steps or they crawl. One global learning rate has to compromise, so training is slow and painfully sensitive to how you set it.

The fix that took over the field is adaptive optimization: give each weight its own effective step size, tuned automatically from the history of its own gradients. Two ideas combine to make this work.

The first is momentum. Instead of stepping purely on the current gradient, you keep a running average of recent gradients and step on that. Picture a ball rolling downhill: it doesn't stop and re-decide at every point; it accumulates velocity, rolling smoothly through small bumps and shallow local dips that would trap a jittery step-by-step walker. Momentum smooths the noisy, batch-to-batch gradients into a steadier direction.

The second is per-weight scaling. The optimizer keeps a running estimate of how large each weight's gradients have typically been, and divides the step by that. Weights that consistently get huge gradients get their steps shrunk; weights that get tiny gradients get their steps amplified. Every weight ends up moving at a sensible pace regardless of the terrain it's on.

Adam: the workhorse

Adam - short for "adaptive moment estimation," introduced by Diederik Kingma and Jimmy Ba in 2014 - combines exactly these two ideas. It tracks a running average of the gradient (momentum, the "first moment") and a running average of the gradient's square (the typical magnitude, the "second moment"), and uses them together to produce a well-scaled, momentum-smoothed update for every weight. The practical payoff is that Adam mostly just works: it converges quickly and tolerates a wide range of learning-rate settings, which is why it became the default for training everything from small classifiers to giant language models. When you read that a model was "trained with Adam," this is the machinery doing the work.

AdamW: the fix that mattered

AdamW, from Ilya Loshchilov and Frank Hutter in 2017, looks like a footnote and is actually a big deal. The issue is weight decay, a standard regularization trick that gently pulls weights toward zero to prevent overfitting. In the original Adam, weight decay got folded into the gradient and then run through Adam's per-weight scaling - which meant the amount of decay each weight received got distorted by that weight's gradient history. Loshchilov and Hutter showed this was a bug in disguise and decoupled weight decay: apply it as a separate, clean shrink on the weights, outside the adaptive machinery. The result generalizes better, and AdamW is now the standard optimizer for large-model training. It is a clean lesson in how a subtle interaction between two components can quietly cost you accuracy until someone untangles it.

Is there anything better than AdamW?

People keep trying. Optimizers like Muon and Shampoo use richer information about how weights interact to compute smarter steps, and each arrives claiming to beat AdamW. But a large 2026 benchmark called OmniOpt put more than two dozen optimizers through a controlled bake-off across model sizes and found that no challenger cleanly dethrones AdamW - an optimizer's edge depends heavily on scale, task, and tuning budget. This is the "no free lunch" reality: there is no universally best optimizer, only best-for-your-situation ones, and a well-tuned AdamW remains a very hard baseline to beat. The choice of optimizer is not a solved footnote; at frontier scale, a few percent of training efficiency is worth millions of dollars, which is exactly why the search continues.

Ground Truth. (2026, July 7). Optimizers: how Adam and AdamW turn gradients into learning. Ground Truth. https://groundtruth.day/learn/optimizers-adam-adamw-and-beyond.html

@misc{groundtruth:optimizers-adam-adamw-and-beyond,
  title  = {Optimizers: how Adam and AdamW turn gradients into learning},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/optimizers-adam-adamw-and-beyond.html}
}

Topics: optimizers · training · adamw · adam · gradient-descent