optimization
Input-Convex Neural Networks: Learning Shapes That Have No Bad Local Minima Lesson
An input-convex neural network is built so its output is a convex function of its input, which means it has one valley, a usable gradient and clean guarantees; Apple used one to learn the best-match score of a cluster of vectors.
Weight initialization: why neural networks start with carefully scaled randomness Lesson
Weight initialization gives a neural network its starting parameters, using carefully scaled randomness to break symmetry and keep signals and gradients usable before learning begins.
Linear regression and least squares: learning a numerical prediction from weighted evidence Lesson
Linear regression predicts a number from weighted inputs, while least squares fits the weights by penalizing large prediction errors and leaves causal interpretation to a separate argument.
Machine-learning compilers: turning tensor operations into efficient device programs Lesson
Machine-learning compilers transform model computations into device programs, using graph optimization and specialization to reduce execution overhead and data movement.
doubleAI introduces Maxwell to rewrite running programs, with proof claims still under scrutiny News
doubleAI says Maxwell replaces running binary code with faster equivalents and reports a 4K video pipeline falling from 250 to 134 seconds.
Expectation-maximization: learning when part of the data is hidden Lesson
Expectation-maximization alternates estimating hidden assignments and refitting model parameters, allowing statistical models to learn from incomplete observations.
Multi-task learning: when shared skills help, and when tasks fight Lesson
Multi-task learning trains one system on several objectives so tasks can share representations, while negative transfer occurs when that sharing hurts a task.
Dream-RSI improves a coding agent’s search policy without changing its weights News
Dream-RSI uses replayed discovery histories to improve the exploration policy around a fixed coding agent, reporting large search-efficiency gains while stopping well short of autonomous model-weight self-modification.
Gradient accumulation: training with a large batch when it will not fit Lesson
Gradient accumulation adds gradients from several small forward-and-backward passes before updating parameters, letting a training job approximate a larger batch without storing that larger batch in memory at once.
Batch normalization: grading every layer on a curve so deep networks train faster Lesson
Batch normalization standardizes each layer's intermediate values using the current training batch's average and spread, then lets the network learn its own scale and offset; introduced in 2015, it let an image classifier reach the same accuracy in 14 times fewer training steps.
Gradient checkpointing: throwing work away so training fits in memory Lesson
Gradient checkpointing cuts the memory a neural network needs during training by deliberately discarding most intermediate results and recomputing them later, trading roughly 30% extra compute for a memory footprint that drops from linear in network depth to the square root of it.
Gradient clipping: the one-line fix that keeps big models from blowing up mid-training Lesson
Exploding gradients happen when the correction signal in a neural network grows enormous on a single unlucky batch, and one huge update destroys weights that took days to learn. Gradient clipping caps the size of that update, keeping the direction and throwing away the magnitude, and it is why frontier training runs survive at all.
Bayesian optimization: how to search when every guess is expensive Lesson
Bayesian optimization is a strategy for finding the best setting of something when each trial is slow or costly: build a cheap statistical model of what you have learned so far, use it to predict both the value and the uncertainty of every untried option, then test whichever option best balances promise against ignorance.
Curriculum learning: why the order of the examples matters Lesson
Training a model on easy examples before hard ones can make it learn faster and end up better than showing the same data in random order, which is why frontier labs now build ladders of increasingly difficult tasks.
Learning rate schedules and warmup Lesson
A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.
Loss of plasticity: why a network that trains forever stops being able to learn Lesson
Keep training a neural network on a stream of new tasks and it does not just forget the old ones - it gradually loses the ability to learn new ones at all, until it performs worse than a freshly initialized network.
What Is Gradient Descent? Lesson
Gradient descent is the optimization method that trains almost every modern neural network: it repeatedly nudges the model's parameters a small step in the direction that most reduces its error, until the error stops falling.
QORL Tool
Open research harness for using a fine-tuned language model to propose and measure Postgres plan hints for recurring analytics.
AutoSaddler Tool
Microsoft's released framework for automatically improving an agent harness from its own failure traces. It diagnoses failed runs, generates structured patches to prompts, tool configurations and control logic, and keeps only patches that survive held-out validation. Reported gains of 9 to 10 points on GAIA2, SWE-Bench Pro and Terminal-Bench 2.0 without touching model weights.