Learn · Beginner

Naive Bayes: classifying by multiplying the evidence

Naive Bayes is a classifier that predicts a category by combining how common that category is with how likely the observed features are within it. Its simplifying assumption is that the features are independent once the category is known. That makes it fast to train and useful as an interpretable baseline, while also making its probability estimates vulnerable to double-counted evidence.

Suppose a system sorts messages into urgent and routine queues. Before reading a message, it knows the fraction of past messages labeled urgent. That is the class prior. It then asks how frequently each observed word appeared in urgent messages and in routine ones. A word associated with urgency raises the urgent score; a word associated with ordinary administration raises the routine score.

The underlying rule is Bayesian updating, but Naive Bayes turns it into a specific learned classification procedure. For a class and a collection of features, its score is the class prior multiplied by the likelihood of each feature given that class. Normalizing the class scores produces probabilities under the fitted model. Choosing the largest score produces the predicted label.

An analogy is an investigator comparing two folders. One folder contains past urgent cases and the other routine cases. Each new clue adds evidence to one folder or the other. The naive shortcut is treating the clues as separate witnesses. If two witnesses are merely repeating the same story, the apparent support becomes stronger than the actual information warrants.

Conditional independence is narrower than saying features never correlate. Words can be correlated overall because the message category influences both. The assumption says that, within a fixed category, knowing one feature adds no information about another. Real language violates this constantly: phrases, topics, and repeated ideas link words together. The method remains useful because getting the highest-scoring class right can be easier than estimating its probability precisely.

For text, the event model matters. A Bernoulli model represents whether each word occurs, including evidence from absence. A multinomial model represents token counts, allowing repeated words to affect the score. Andrew McCallum and Kamal Nigam’s 1998 comparison of event models is a canonical treatment of why these choices produce different text classifiers. Neither should be confused with a modern language model that predicts a sequence using a rich representation of context.

Training mostly means counting labeled examples and feature occurrences, then converting counts into probability estimates. Small datasets create a practical problem: a word absent from one class receives an estimated likelihood of zero. Multiplying by zero wipes out that class score, even when the absence is merely a sampling accident. Additive smoothing supplies small pseudo-counts before normalization, reserving some probability for unseen events.

Implementations usually add log-probabilities instead of multiplying many tiny probabilities. The logarithm preserves the ranking while preventing numerical underflow. The resulting computation is a class-specific intercept plus a sum of feature contributions. That can look similar to a linear classifier, but the route used to learn those contributions is different.

Logistic regression directly learns the conditional probability of the label given the features. Naive Bayes instead specifies how a class and its features could jointly arise, then uses Bayes’ rule to infer the class. This is the generative-versus-discriminative distinction in a concrete setting. Generative here describes a probability model of observations; it does not mean the classifier is a chatbot or a picture generator.

Confidence needs a separate check. If urgent messages often contain both parts of the phrase payment failed, treating those clues as independent can overstate the evidence. The predicted label might be right while the reported probability is too extreme. Ground Truth’s calibration lesson explains how to test whether confidence matches observed frequency. A decision threshold should reflect error costs and review capacity rather than automatically defaulting to one half.

Evaluation also needs an honest holdout set, a training-only vocabulary, and representative class frequencies. Learning the vocabulary or tuning smoothing on the test set leaks information into the evaluation. A rare urgent class can also make overall accuracy look good for an unhelpful system that mostly predicts routine.

Today’s LFM3 release makes small decision components topical, but a neural decision model and Naive Bayes are different mechanisms. The useful connection is the discipline of testing a simple baseline before accepting a complex deployment. Naive Bayes is cheap, explicit, and limited. Understanding exactly how it combines evidence helps explain both why simple classifiers work and why repeated evidence should not automatically produce greater trust.

Ground Truth. (2026, October 28). Naive Bayes: classifying by multiplying the evidence. Ground Truth. https://groundtruth.day/learn/naive-bayes-generative-classification.html

@misc{groundtruth:naive-bayes-generative-classification,
  title  = {Naive Bayes: classifying by multiplying the evidence},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/naive-bayes-generative-classification.html}
}

Topics: fundamentals · classification · probability · naive-bayes · supervised-learning