hyperparameters
Everything on Ground Truth tagged “hyperparameters” — 2 items.
Bayesian optimization: how to search when every guess is expensive Lesson
Bayesian optimization is a strategy for finding the best setting of something when each trial is slow or costly: build a cheap statistical model of what you have learned so far, use it to predict both the value and the uncertainty of every untried option, then test whichever option best balances promise against ignorance.
Learning rate schedules and warmup Lesson
A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.