fundamentals

Everything on Ground Truth tagged “fundamentals” — 88 items.

Beam search: keeping several drafts alive to find a likelier sentence Lesson

Beam search is a decoding method that keeps the few highest-scoring partial outputs at every step instead of committing to the single likeliest word, which finds more probable sentences than greedy decoding and exposes why the most probable text is often not the best text.

Batch normalization: grading every layer on a curve so deep networks train faster Lesson

Batch normalization standardizes each layer's intermediate values using the current training batch's average and spread, then lets the network learn its own scale and offset; introduced in 2015, it let an image classifier reach the same accuracy in 14 times fewer training steps.

Process reward models: grading the working, not just the answer Lesson

A process reward model scores each step of a model's reasoning rather than only the final answer, which makes it possible to catch a wrong turn even when the conclusion happens to come out right - and to tell a lucky guess apart from genuine reasoning.

Continuous batching and paged attention: how one GPU serves a thousand people at once Lesson

Continuous batching and paged attention are the two scheduling tricks that let a single GPU serve many users at once instead of one at a time, and together they are responsible for most of the collapse in the price of running a language model.

Pretraining: how a model learns almost everything it knows before anyone teaches it anything Lesson

Pretraining is the phase where a model reads an enormous quantity of text and learns only to predict the next word, and it is where essentially all of a language model's knowledge and fluency comes from. Everything afterwards adjusts behaviour, not knowledge.

Emergent abilities: do models suddenly gain skills, or are we measuring badly? Lesson

Emergent abilities are skills that appear absent in smaller models and present in larger ones, seeming to switch on abruptly rather than improve gradually. A influential 2023 rebuttal argued the sharpness is largely an artifact of all-or-nothing scoring, and the honest answer is that both sides are partly right.

Property-based testing: test the rule, not just the examples Lesson

Property-based testing generates many inputs automatically and checks general rules that software should always obey, such as round trips, invariants, and equivalence under harmless transformations; it finds edge cases that example-by-example unit tests often miss.

Program synthesis: making a computer write the program from the specification Lesson

Program synthesis is the task of automatically constructing a program that satisfies a specification such as examples, types, tests, or logical constraints; it matters because a verifiable specification can turn programming from writing every instruction into searching for a correct implementation.

Gradient checkpointing: throwing work away so training fits in memory Lesson

Gradient checkpointing cuts the memory a neural network needs during training by deliberately discarding most intermediate results and recomputing them later, trading roughly 30% extra compute for a memory footprint that drops from linear in network depth to the square root of it.

Gradient clipping: the one-line fix that keeps big models from blowing up mid-training Lesson

Exploding gradients happen when the correction signal in a neural network grows enormous on a single unlucky batch, and one huge update destroys weights that took days to learn. Gradient clipping caps the size of that update, keeping the direction and throwing away the magnitude, and it is why frontier training runs survive at all.

Cross-validation: how you find out whether a model learned anything or just memorised the answers Lesson

A model's score on the data it trained on tells you nothing about whether it will work, because memorising is easier than learning. Cross-validation and holdout sets solve this by measuring the model only on examples it has never seen, and the discipline of keeping a final test set untouched is what separates a real result from a self-flattering one.

Benchmark contamination: when the test is already in the training data Lesson

Benchmark contamination is what happens when the questions used to evaluate a model were in the data used to train it, turning a test of reasoning into a test of memory and inflating scores in ways that are hard to detect after the fact.

Active learning: letting the model choose what to label next Lesson

Active learning is a training strategy in which the model picks which unlabeled examples should be labeled next, choosing the ones it is most uncertain about so that a fixed labeling budget buys the most improvement possible.

N-gram language models Lesson

An n-gram language model predicts the next word by counting how often each word followed the previous one or two words in a large corpus -- the simplest working language model there is, and the direct ancestor of everything that came after.

The bias-variance tradeoff: why a model can fail by being too simple or too clever Lesson

Every model's error splits into two opposing parts -- bias, from being too rigid to capture the pattern, and variance, from being so flexible it memorizes noise. Reducing one usually raises the other, and the whole craft of machine learning is finding where their sum is smallest.

Inference cost and token economics: why output tokens cost more than input Lesson

Model providers charge separately for the tokens you send and the tokens the model writes, and output is typically three to five times more expensive. The reason is architectural: input is processed in one parallel pass, while every output token requires its own full pass through the model.

Teacher forcing and exposure bias Lesson

Teacher forcing trains a sequence model by always feeding it the correct previous token instead of its own output, which makes training fast and stable but leaves the model unprepared for its own mistakes at generation time -- a mismatch called exposure bias.

Pseudo-labeling and self-training Lesson

Pseudo-labeling is training a model on labels it produced itself: run a model over unlabeled data, keep the predictions it is confident about, and treat them as ground truth for the next round of training. It works surprisingly well, and it fails in one specific way -- by confidently reinforcing its own mistakes.

Out-of-distribution detection: teaching a model to say I have not seen this before Lesson

Out-of-distribution detection is the problem of getting a model to flag inputs unlike its training data instead of confidently guessing, and it has quietly moved from an image-classifier safety concern to core infrastructure for monitoring AI agents in production.

Activation steering: changing a model's behaviour by editing its thoughts Lesson

Activation steering changes what an AI model does by adding or subtracting a direction from its internal numbers while it runs, with no retraining and no prompt changes, and the fact that it works at all says something uncomfortable about how safety training is stored.

Training data attribution: which examples actually made the model do that? Lesson

Training data attribution is the set of techniques for tracing a model's output back to the specific training examples responsible for it, and the honest state of the art is that it often fails.

Evaluation awareness: when the model can tell it is being tested Lesson

Evaluation awareness is a model's ability to detect that it is being tested rather than used, and to behave differently as a result, which quietly undermines the safety evaluations that are supposed to catch exactly that behavior.

De novo protein design: how a computer invents a molecule that sticks Lesson

De novo protein design means inventing a protein that does not exist in nature and that binds a chosen target, by predicting how a chain of amino acids will fold and then searching for a chain that folds into a shape which fits. It works because structure prediction got good enough to grade designs before anyone builds them.

Bayesian optimization: how to search when every guess is expensive Lesson

Bayesian optimization is a strategy for finding the best setting of something when each trial is slow or costly: build a cheap statistical model of what you have learned so far, use it to predict both the value and the uncertainty of every untried option, then test whichever option best balances promise against ignorance.

ROC curves and AUC: the number behind 'the detector works' Lesson

AUC measures how well a detector ranks positives above negatives across every possible threshold, which is why 0.5 means coin flip and why a high AUC can still hide a useless system.

Data augmentation: teaching a model more without collecting more Lesson

Data augmentation multiplies a training set by transforming existing examples in ways that change the input but not the answer, teaching a model which differences to ignore -- and the choice of transformation encodes exactly what you want it to be blind to.

Federated learning Lesson

Federated learning trains a shared model across many devices without collecting their data, by sending the model to the data and averaging the resulting updates instead.

Ablation studies Lesson

An ablation study removes one piece of a system and re-measures it, which is how researchers show a component earns its place instead of just riding along.

Encrypted inference: can a model answer a question it cannot read? Lesson

The two competing ways to run AI on data the server is not supposed to see: sealed hardware enclaves, which ship today and are fast, and homomorphic encryption, which is mathematically stronger and still far too slow.

Curriculum learning: why the order of the examples matters Lesson

Training a model on easy examples before hard ones can make it learn faster and end up better than showing the same data in random order, which is why frontier labs now build ladders of increasingly difficult tasks.

Multi-agent systems: what changes when agents stop being tools to each other Lesson

A multi-agent system is one where several AI agents act at the same time in a shared environment, and the interesting failures come not from any single agent being wrong but from many agents being identically right.

Encoder, decoder, or both: the three ways to build a transformer Lesson

The same transformer block can be assembled three ways -- encoder-only, decoder-only, or encoder-decoder -- and the choice determines whether a model reads text, writes text, or converts one into the other.

Bayesian updating: how a belief should change when evidence arrives Lesson

Bayesian updating is the rule for revising a belief when new evidence comes in: start with a prior, weigh how much more likely the evidence is under one hypothesis than another, and get a posterior.

Sample complexity: how many examples does learning actually take? Lesson

Sample complexity is the number of training examples a learning method needs to reliably learn a given pattern. It is the formal way of asking why a child learns a word from three exposures while a language model needs the internet, and it is what separates a method that scales from one that merely works.

Learning rate schedules and warmup Lesson

A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.

In-context learning Lesson

In-context learning is a language model's ability to pick up a new task from examples placed in its prompt, without any change to its weights, and it is the reason prompting works at all.

Loss of plasticity: why a network that trains forever stops being able to learn Lesson

Keep training a neural network on a stream of new tasks and it does not just forget the old ones - it gradually loses the ability to learn new ones at all, until it performs worse than a freshly initialized network.

Chat templates: the invisible tags that tell a model who is speaking Lesson

A language model never sees a conversation - it sees one long string, and small marker tokens are the only thing telling it which parts are your instructions, which are its own thoughts, and which are untrusted data from the outside world.

Genome language models: predicting DNA the way a chatbot predicts text Lesson

Genome language models are trained to predict the next base in a DNA sequence, the same objective that produces chatbots, and they turn out to learn enough biology to write working viral genomes from scratch.

Ensembles: why averaging several models beats picking the best one Lesson

An ensemble runs several models on the same input and combines their answers, which reliably beats any single member because independent mistakes cancel while correct signal adds -- and it is where forecast probabilities come from.

Shortcut learning: when a model gets the right answer for the wrong reason Lesson

Shortcut learning is what happens when a model finds a cue that correlates with the right answer but has nothing to do with the actual task, and uses it instead of learning the thing you wanted.

Credit assignment: figuring out which step deserved the reward Lesson

Credit assignment is the problem of working out which of an agent's many decisions actually caused the eventual outcome, and it is the central difficulty in training anything that acts over a long sequence of steps.

Differential privacy: a mathematical promise about what a model can reveal about you Lesson

Differential privacy is a formal guarantee that a system's output would look almost the same whether or not any single person's data was included, enforced by adding carefully calibrated noise. It is the only privacy protection for machine learning that comes with a provable bound rather than a hope.

Autoencoders and VAEs: teaching a model to compress the world Lesson

An autoencoder is a network trained to squeeze data through a narrow bottleneck and rebuild it, and a variational autoencoder makes that bottleneck a smooth space of probabilities you can sample from, which is why nearly every modern image and video generator does its work inside one.

Model merging: combining two fine-tuned models by averaging their weights Lesson

Model merging combines two or more models that share a common ancestor by arithmetic on their weights - averaging them, adding their differences, or interpolating between them - producing a single model with both sets of skills and no additional training.

Guardrail models: the second AI that decides whether the first one's answer ships Lesson

A guardrail model is a small separate classifier that reads what a user sends an AI and what the AI sends back, then scores whether it violates a policy - so safety becomes a component the operator owns and can inspect, rather than a behaviour buried in the main model's weights.

Self-Play: How a Model Gets Better by Competing With Itself Lesson

Self-play is training a model by having it compete or cooperate against copies of itself, so the opponent gets harder exactly as fast as the learner improves and the training data never runs out.

Full-Duplex Speech Models: Listening and Talking at the Same Time Lesson

A full-duplex speech model processes incoming audio while it is generating outgoing audio, which removes the turn detector that decides when you have stopped speaking and makes interruption, backchannels, and overlap possible.

Calibration: whether a model's confidence means anything Lesson

A model is calibrated when the things it says with 70% confidence turn out to be true about 70% of the time - and calibration is a completely separate property from accuracy, which is why a model can be both frequently right and systematically untrustworthy about when it is wrong.

BM25 and lexical search: the keyword formula that keeps beating neural retrieval Lesson

BM25 is a decades-old formula that ranks documents by how often a query's rare words appear in them, adjusted for document length - and it remains the baseline that modern AI retrieval systems have to beat, often unsuccessfully.

Vision Transformers: what happens when you feed a picture to a language architecture Lesson

A Vision Transformer chops an image into a grid of small patches, treats each patch as a word, and runs the exact same Transformer machinery that powers language models over the resulting sequence. Google Research showed in 2020 that this beats purpose-built image networks once you train it on enough data, and it is why today's image, video and robot models all share one architecture.

Data poisoning and backdoors: attacking a model through what it eats Lesson

Data poisoning is an attack that corrupts a model by tampering with its training data rather than its code, and a backdoor is the sharpest form: a model that behaves perfectly until it sees a secret trigger. Anthropic and the UK AI Safety Institute found in 2025 that just 250 poisoned documents compromised models from 600 million to 13 billion parameters alike, which means scale does not dilute the threat.

The harness: the code around a model that decides how smart it looks Lesson

An agent harness is the ordinary software wrapped around a language model that decides what it sees, what tools it can call, and what it remembers between steps, and changing it can swing benchmark scores several times over without touching the model at all.

Catastrophic forgetting: why teaching an AI something new can erase what it knew Lesson

Catastrophic forgetting is the tendency of a neural network to lose an old skill when trained on a new one, because both skills are stored in the same weights and nothing protects the old settings.

Forward and Reverse KL Divergence Lesson

KL divergence measures how far one probability distribution is from another, and it is asymmetric: swapping which distribution goes first changes whether your model tries to cover everything the target does or commit to one part of it.

Classifier-Free Guidance Lesson

Classifier-free guidance is the knob that makes image and video models actually follow your prompt: it runs the model twice - once with your prompt, once without - and pushes the output along the difference.

Jailbreaking and red-teaming: breaking an AI on purpose, before someone else does Lesson

A jailbreak is an input that makes a model do what its training told it to refuse, and red-teaming is the organized practice of hunting for those inputs deliberately, which is how every safety claim about a model gets tested before release.

How a model is stored: safetensors, GGUF, and why one model arrives in 96 files Lesson

A trained model is just a large dictionary of numbered arrays saved to disk, and the file format that holds them determines whether the model loads safely, loads fast, and loads at all on your hardware.

Why AI Inference Runs Out of Memory Bandwidth Before It Runs Out of Math Lesson

Generating text with a language model is limited by how fast weights can be moved from memory into the processor, not by how fast the processor can multiply, which is why most of a GPU sits idle during inference.

Neural text-to-speech: how a model turns writing into a voice Lesson

Neural text-to-speech converts written text into audio in three stages - working out the sounds, deciding how long each one lasts, and generating the actual waveform - and the last stage, the vocoder, is where most of the model's size and difficulty hides.

Convolutional neural networks: how machines learned to see Lesson

A convolutional neural network learns small reusable filters that slide across an image, so the same edge or texture detector works anywhere in the frame - the idea that made computer vision practical and still runs inside modern image, audio and video systems.

Training data deduplication: why the same text twice makes a model worse Lesson

Deduplication is the process of finding and removing repeated or near-repeated documents from a training corpus, and it reliably makes language models better, cheaper, and safer - the same text seen many times gets memorised rather than learned from.

Activation functions: the tiny nonlinearity that makes deep learning possible Lesson

An activation function is a simple nonlinear operation applied to each number flowing through a neural network, and without one, stacking a hundred layers would be mathematically identical to using a single layer. Modern transformers mostly use gated variants like SwiGLU, which Noam Shazeer showed outperform plain ReLU.

Mixed-precision training: why models are trained in half-broken numbers on purpose Lesson

Modern models are trained using 16-bit and even 8-bit numbers instead of the 32-bit standard, roughly doubling speed and halving memory, by carefully keeping full precision exactly where the arithmetic would otherwise fall apart.

Distributed training: how one model gets split across thousands of chips Lesson

No single chip can hold a frontier model, so training is split across thousands of them in four distinct ways -- by data, by layer, by tensor, and by expert -- and choosing the right mix is what separates a cluster running at a third of its potential from one running at a tenth.

Softmax and Cross-Entropy: How a Model Turns Scores Into a Confident Guess Lesson

Softmax turns a model's raw scores into a probability distribution, and cross-entropy measures how far that distribution is from the true answer; together they are the training target behind almost every classifier and language model.

RNNs and LSTMs: How Neural Networks Learned to Remember, Before Transformers Lesson

Recurrent neural networks process sequences one step at a time while carrying a memory, and LSTMs added gates to fix their forgetfulness; together they powered a decade of sequence modeling and set up exactly the problem transformers later solved.

Markov Decision Processes: The Math Behind How AI Learns to Act Lesson

A Markov Decision Process is the mathematical framework that formalizes sequential decision-making -- states, actions, rewards, and transitions -- and it is the foundation on which nearly all reinforcement learning, from game-playing AI to robot control, is built.

Adversarial Examples: Fooling AI With Changes You Can't See Lesson

Adversarial examples are inputs deliberately perturbed by tiny, often invisible amounts that cause an AI model to make confident mistakes, revealing that neural networks rely on fragile patterns rather than robust understanding.

Regularization: why deliberately handicapping a model makes it better Lesson

Regularization is the family of techniques that make a model perform worse on its training data in order to make it perform better on data it has never seen, with dropout and weight decay the two that shaped modern deep learning.

Perplexity: the number that tells you a model still works Lesson

Perplexity measures how surprised a language model is by real text, expressed as the number of words it was effectively choosing between at each step -- lower means less confused, and a jump from single digits into the thousands means the model is broken.

Layer normalization: the rescaling that keeps deep networks trainable Lesson

Layer normalization rescales the numbers flowing through each layer of a network to a consistent range, which stops signals from exploding or vanishing as they pass through dozens of layers -- and it is what makes training a transformer at all possible.

Diffusion models: how AI turns noise into images and video Lesson

Diffusion models generate images and video by starting from pure random noise and removing it step by step until a coherent picture emerges -- the technique behind Stable Diffusion, Sora, and interactive video systems like Vidu S1.

Tool Use and Function Calling: How LLMs Act on the World Lesson

Tool use, also called function calling, is how a language model stops being a text box and starts doing things: it emits a structured request to call an external function, your code runs it, and the result is fed back so the model can continue reasoning.

GANs: the two-network duel that taught AI to imagine Lesson

A generative adversarial network trains two neural networks against each other -- a forger trying to create fake data and a detective trying to spot it -- until the forger's output becomes indistinguishable from the real thing, the breakthrough that first made AI image generation convincing.

Flow matching: how AI learns to turn noise into a picture Lesson

Flow matching teaches an AI to generate images by learning a smooth flow that carries random noise, step by step, into a realistic picture -- a cleaner, faster successor to diffusion that powers modern image models like FLUX.

What Is Gradient Descent? Lesson

Gradient descent is the optimization method that trains almost every modern neural network: it repeatedly nudges the model's parameters a small step in the direction that most reduces its error, until the error stops falling.