Learn · Intermediate

Out-of-distribution detection: teaching a model to say I have not seen this before

Out-of-distribution detection is the problem of getting a model to recognize when an input is unlike anything it was trained on, so it can say "I don't know" instead of confidently guessing. It matters because neural networks fail silently by default: show an image classifier trained on animals a photograph of a car, and it will not object, it will tell you it is 94 percent confident the car is a cat. The model has no built-in notion of "outside my experience," and building one is harder than it sounds.

The reason this is not automatic comes down to how classifiers are constructed. A model trained to sort inputs into ten categories will sort everything into one of those ten categories, because that is the only vocabulary it has. The softmax layer at the end normalizes its scores into probabilities that sum to one, which means the model is structurally incapable of expressing "none of the above." It can only express relative preference among options it already knows.

An analogy: a wine expert who has only ever tasted French wines, asked to name the region of a glass of orange juice. Nothing in their training produces the answer "this is not wine." They will confidently say Burgundy.

The baseline that turned out to be hard to beat

The field's starting point is a 2016 paper by Dan Hendrycks and Kevin Gimpel, A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Their proposal was almost embarrassingly simple: look at the highest probability the softmax produces. If the model's best guess is 0.99, it is probably in familiar territory. If the best guess is 0.31, something may be wrong. Threshold on that number.

This works better than it has any right to, and it remains the honest baseline against which everything else is measured. It also fails in a specific and important way: modern networks are systematically overconfident, a problem covered separately under calibration. A network can be 99 percent confident about noise. So the signal is real but noisy, and a lot of subsequent work is about extracting a cleaner version of it.

Four families of improvement

Better use of the output scores. ODIN, from Shiyu Liang and colleagues in this 2017 paper, sharpened the baseline with two tricks: temperature scaling to spread out the probability distribution, and a small adversarial perturbation of the input that increases the softmax score more for in-distribution inputs than for outliers. Later, Energy-based Out-of-distribution Detection by Weitang Liu and colleagues argued that the softmax throws away useful information, and that an energy score computed from the raw logits separates in-distribution from out-of-distribution data more cleanly than the maximum probability does.

Distance in feature space. Rather than reading the output, look at where the input lands in the model's internal representation. Kimin Lee and colleagues proposed in A Simple Unified Framework for Detecting Out-of-Distribution Samples fitting a Gaussian to each class in feature space and measuring Mahalanobis distance to the nearest one. Familiar inputs land near a class cluster; unfamiliar inputs land in empty space. This is the direct ancestor of every "off-manifold" score in modern systems.

Training on outliers. Hendrycks and colleagues showed in Deep Anomaly Detection with Outlier Exposure that if you have access to some out-of-distribution data during training -- any broad, cheap, unrelated dataset -- you can train the model to output a uniform distribution on it. The model generalizes the habit of being uncertain to outliers it never saw, which is more useful than it sounds.

Ensembles and disagreement. If you train several models and they agree confidently, the input is probably familiar. If they disagree, it probably is not. See ensembles and why averaging predictions works.

Why it is having a moment

Out-of-distribution detection began as an image-classification safety problem and has quietly become core infrastructure for AI oversight. Any system that monitors an AI in production needs some way to say "this input, or this internal state, is unlike normal traffic." That is the same question.

A concrete 2026 example: research on agents coordinating through a channel that transcripts never see builds a monitor whose first of three signals is exactly an off-manifold score on internal states -- fit on benign traffic only, with attack examples held back for evaluation. That training discipline is the field's hard-won standard: a detector trained on the anomalies it will later be scored against tells you almost nothing about anomalies you have not imagined.

The honest limits

Out-of-distribution detection has a definitional problem it has never solved. "Out of distribution" is not a property of an input; it is a relationship between an input and a training set, and the boundary is fuzzy in ways that resist formalization. Near-distribution outliers -- a dog breed the model was not trained on -- are far harder than far-distribution ones, and most reported numbers use far-distribution benchmarks that flatter the methods.

There is also an unavoidable trade-off. Every detector has a threshold, and moving it trades false alarms against missed detections. Set it tight and you reject valid inputs; set it loose and unfamiliar inputs sail through. Which error is worse is a question about your application, not about your model, and no amount of method development answers it for you. The tooling for reasoning about that trade-off is covered under ROC curves and AUC.

Ground Truth. (2026, August 20). Out-of-distribution detection: teaching a model to say I have not seen this before. Ground Truth. https://groundtruth.day/learn/out-of-distribution-detection.html

@misc{groundtruth:out-of-distribution-detection,
  title  = {Out-of-distribution detection: teaching a model to say I have not seen this before},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/out-of-distribution-detection.html}
}

Topics: evaluation · safety · uncertainty · monitoring · fundamentals