statistics
Bradley–Terry preference models: turning head-to-head choices into scores Lesson
Bradley–Terry models summarize pairwise choices as scores and help train AI reward models, but a single ranking can hide disagreement and task differences.
Markov chain Monte Carlo: exploring a probability distribution one step at a time Lesson
Learn how dependent random walks approximate complex probability distributions and why convergence checks matter.
Sequential testing: how to watch a changing model without chasing noise Lesson
Sequential testing plans repeated checks so that early stopping does not turn ordinary noise into a false discovery.
Paired comparisons: how to tell whether one AI system really beats another Lesson
Paired comparisons test AI systems on the same examples and use task-level differences to make a more informative comparison.
Simpson’s paradox: when the overall AI score reverses every group result Lesson
Simpson’s paradox shows how different group proportions can reverse an overall comparison even when the within-group results point the other way.
Inter-rater reliability: when two graders disagree Lesson
Inter-rater reliability measures how consistently graders apply an evaluation, and Cohen’s kappa adjusts agreement for label frequencies.
Bootstrap resampling: estimating uncertainty by repeatedly rebuilding your sample Lesson
The bootstrap estimates sampling uncertainty by resampling observed units with replacement and recomputing a statistic, while preserving the structure that makes those units independent.
Importance sampling: evaluating a new AI policy from an old policy’s logs Lesson
Importance sampling reweights logged outcomes by how differently a target policy would choose their actions, enabling bounded off-policy evaluation when coverage and logging assumptions hold.
Causal graphs: deciding what to adjust for before interpreting an AI result Lesson
Causal graphs encode assumptions about cause and effect so researchers can distinguish useful statistical adjustment from adjustments that introduce bias.
Conformal prediction: turning model uncertainty into answer sets with coverage guarantees Lesson
Conformal prediction uses held-out examples to construct answer sets or intervals with a finite-sample coverage target under exchangeability assumptions.
Expectation-maximization: learning when part of the data is hidden Lesson
Expectation-maximization alternates estimating hidden assignments and refitting model parameters, allowing statistical models to learn from incomplete observations.
Randomized trials: measuring what an AI tool actually changes Lesson
Randomized trials estimate an AI tool’s causal effect by separating access to the tool from preexisting differences among its users.
Principal component analysis: finding the directions that summarize your data Lesson
Principal component analysis compresses a dataset into perpendicular directions that preserve the most variation, without guaranteeing useful or causal features.
Preregistration: deciding what counts as evidence before an AI test runs Lesson
Preregistration records an evaluation’s hypotheses, metrics, exclusions, and decision rules before results can influence them.
Equivalence testing: how to show two AI systems are close enough Lesson
Equivalence testing asks whether a measured difference is small enough to be practically unimportant, which is stronger and more useful than merely failing to find a statistically significant difference.
N-gram language models Lesson
An n-gram language model predicts the next word by counting how often each word followed the previous one or two words in a large corpus -- the simplest working language model there is, and the direct ancestor of everything that came after.
The bias-variance tradeoff: why a model can fail by being too simple or too clever Lesson
Every model's error splits into two opposing parts -- bias, from being too rigid to capture the pattern, and variance, from being so flexible it memorizes noise. Reducing one usually raises the other, and the whole craft of machine learning is finding where their sum is smallest.
Null baselines and multiple comparisons: why an untrained model can look like it learned Lesson
A null baseline is what your measurement reports when nothing happened, and it is almost never zero. Without measuring it, and without correcting for how many things you tested at once, an improvement that is pure noise will look exactly like a real result.
A frozen model can look like it taught itself, and most self-improvement results never checked News
A new audit ran a completely untrained control model through the same self-training pipeline as the real thing and found it appeared to both learn and forget, meaning most reported self-improvement gains are measurement artifacts unless the null was measured too.
Bayesian updating: how a belief should change when evidence arrives Lesson
Bayesian updating is the rule for revising a belief when new evidence comes in: start with a prior, weigh how much more likely the evidence is under one hypothesis than another, and get a posterior.