adam

Optimizers: how Adam and AdamW turn gradients into learning Lesson

An optimizer is the rule that decides how a neural network changes its weights after each mistake; Adam and its refinement AdamW became the default because they adapt the step size for every weight, making training faster and far less finicky than plain gradient descent.