Learn · Intermediate

Test-Time Training

Test-time training is the practice of updating a model's weights on the specific problem it is being asked to solve, at the moment it is asked, instead of freezing the weights after training and only running a forward pass. Each test example becomes a miniature training run. It is one of the few techniques that reliably closes the gap between a model that has seen a task's general shape and a model that has actually adapted to the instance in front of it.

The idea sounds like a violation of something. We are taught that training and inference are separate phases: you fit the model on training data, you freeze it, you deploy it, and any further learning would be contamination. Test-time training breaks that separation on purpose, and the reason it is legitimate is that the update signal does not come from the answer key. It comes from the input.

Where the free training signal comes from

Consider the original formulation from Yu Sun and colleagues at Berkeley in 2019. They took an image classifier and gave it a second job during training: predict how much a randomly rotated copy of an image had been rotated. That second task is self-supervised -- you generate the label yourself by choosing the rotation, so no human annotation is needed. At test time, when a new image arrives, you cannot check whether your classification is right, but you can still do the rotation task on that image. So you take a few gradient steps on the rotation objective for that one image, which nudges the shared features toward the actual data in front of you, and only then classify. Accuracy under distribution shift improved substantially.

The trick generalizes because a surprising amount of structure in an input is checkable without knowing the answer. Predict a masked-out patch. Reconstruct the input. Predict the next token in the document you were handed. All of these give you gradients without touching the label you are being graded on.

The clearest modern payoff is on ARC-AGI, Francois Chollet's abstract reasoning benchmark. Each ARC puzzle hands you a few example grid transformations and asks you to apply the same rule to a new grid. Ekin Akyurek and colleagues at MIT showed in 2024 that fine-tuning a language model on those bundled demonstrations at test time -- augmented with rotations, reflections, and color permutations -- produced very large improvements over the same model prompted normally. This is exactly what the benchmark asks for. ARC deliberately gives you a rule you have never seen, so a model that cannot adapt to a new rule at inference is structurally disadvantaged, no matter how good its prior. This is why several of the strongest ARC results have come from small task-specific transformers trained from scratch at test time rather than from frontier general-purpose models -- and why comparing the two is subtle, as we noted when a 150-million-parameter model set an ARC-AGI record for cost rather than score.

The mental model

Think of a general model as a doctor with broad training and a test-time-trained model as the same doctor after spending twenty minutes reading this one patient's chart. Nothing about medicine changed. What changed is that the general knowledge got re-weighted toward this case. The cost is that the twenty minutes happen per patient, and the doctor forgets afterward -- the adapted weights are typically thrown away once the answer is produced, because keeping them would mean the model drifts differently for every user.

Mechanically, test-time training almost always uses LoRA or another parameter-efficient fine-tuning method rather than updating everything. You are doing a handful of gradient steps on a handful of examples, so full fine-tuning would both cost too much and overfit immediately. A small adapter, trained for tens of steps and then discarded, is the standard recipe.

Where it fits among neighbors

It is worth separating three ideas that get conflated. In-context learning adapts behavior through the prompt, with weights frozen -- cheap, fast, and limited by what fits in context. Test-time compute buys quality with more inference work: longer reasoning chains, more samples, search over candidates -- also weights-frozen. Test-time training is the third axis, and it is the only one that changes the model. The three compose: you can give a model demonstrations in context, fine-tune it on those demonstrations, and then let it reason at length.

The costs are real. You pay a training run per query, which can be orders of magnitude more expensive than a forward pass, and it destroys the batching efficiency that makes serving cheap -- every user now needs their own weights. Yu Sun's later work on expressive hidden states attacks exactly this by folding the test-time update into the architecture itself, treating a recurrent layer's hidden state as a small model that is trained by the sequence as it streams past. That reframing is one of the more elegant results in recent sequence modeling: an RNN's hidden state and a model being fine-tuned turn out to be the same object viewed two ways.

The honest limitation is that test-time training shines exactly where the test distribution differs sharply from training and where each instance carries its own supervision. Puzzle benchmarks, distribution shift, and personalization fit. General open-ended chat mostly does not, because there is no per-instance objective worth a gradient step. If you are considering it, the first question is not "will this help" but "what would I compute the gradient on?" If you cannot answer that from the input alone, the technique does not apply.

Ground Truth. (2026, September 1). Test-Time Training. Ground Truth. https://groundtruth.day/learn/test-time-training.html

@misc{groundtruth:test-time-training,
  title  = {Test-Time Training},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/test-time-training.html}
}

Topics: test-time-training · fine-tuning · arc-agi · generalization · inference · adaptation