optimization
Batch normalization: grading every layer on a curve so deep networks train faster Lesson
Batch normalization standardizes each layer's intermediate values using the current training batch's average and spread, then lets the network learn its own scale and offset; introduced in 2015, it let an image classifier reach the same accuracy in 14 times fewer training steps.
Gradient checkpointing: throwing work away so training fits in memory Lesson
Gradient checkpointing cuts the memory a neural network needs during training by deliberately discarding most intermediate results and recomputing them later, trading roughly 30% extra compute for a memory footprint that drops from linear in network depth to the square root of it.
Gradient clipping: the one-line fix that keeps big models from blowing up mid-training Lesson
Exploding gradients happen when the correction signal in a neural network grows enormous on a single unlucky batch, and one huge update destroys weights that took days to learn. Gradient clipping caps the size of that update, keeping the direction and throwing away the magnitude, and it is why frontier training runs survive at all.
Bayesian optimization: how to search when every guess is expensive Lesson
Bayesian optimization is a strategy for finding the best setting of something when each trial is slow or costly: build a cheap statistical model of what you have learned so far, use it to predict both the value and the uncertainty of every untried option, then test whichever option best balances promise against ignorance.
Curriculum learning: why the order of the examples matters Lesson
Training a model on easy examples before hard ones can make it learn faster and end up better than showing the same data in random order, which is why frontier labs now build ladders of increasingly difficult tasks.
Learning rate schedules and warmup Lesson
A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.
Loss of plasticity: why a network that trains forever stops being able to learn Lesson
Keep training a neural network on a stream of new tasks and it does not just forget the old ones - it gradually loses the ability to learn new ones at all, until it performs worse than a freshly initialized network.
What Is Gradient Descent? Lesson
Gradient descent is the optimization method that trains almost every modern neural network: it repeatedly nudges the model's parameters a small step in the direction that most reduces its error, until the error stops falling.
AutoSaddler Tool
Microsoft's released framework for automatically improving an agent harness from its own failure traces. It diagnoses failed runs, generates structured patches to prompts, tool configurations and control logic, and keeps only patches that survive held-out validation. Reported gains of 9 to 10 points on GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 without touching model weights.