statistics
Equivalence testing: how to show two AI systems are close enough Lesson
Equivalence testing asks whether a measured difference is small enough to be practically unimportant, which is stronger and more useful than merely failing to find a statistically significant difference.
N-gram language models Lesson
An n-gram language model predicts the next word by counting how often each word followed the previous one or two words in a large corpus -- the simplest working language model there is, and the direct ancestor of everything that came after.
The bias-variance tradeoff: why a model can fail by being too simple or too clever Lesson
Every model's error splits into two opposing parts -- bias, from being too rigid to capture the pattern, and variance, from being so flexible it memorizes noise. Reducing one usually raises the other, and the whole craft of machine learning is finding where their sum is smallest.
Null baselines and multiple comparisons: why an untrained model can look like it learned Lesson
A null baseline is what your measurement reports when nothing happened, and it is almost never zero. Without measuring it, and without correcting for how many things you tested at once, an improvement that is pure noise will look exactly like a real result.
A frozen model can look like it taught itself, and most self-improvement results never checked News
A new audit ran a completely untrained control model through the same self-training pipeline as the real thing and found it appeared to both learn and forget, meaning most reported self-improvement gains are measurement artifacts unless the null was measured too.
Bayesian updating: how a belief should change when evidence arrives Lesson
Bayesian updating is the rule for revising a belief when new evidence comes in: start with a prior, weigh how much more likely the evidence is under one hypothesis than another, and get a posterior.