Learn · Beginner

Softmax and Cross-Entropy: How a Model Turns Scores Into a Confident Guess

Softmax turns a neural network's raw output scores into a set of probabilities that add up to one, and cross-entropy measures how wrong those probabilities are compared to the true answer. Together they form the training target behind nearly every classifier and every language model, including the one predicting the next word in this sentence. If you understand this pair, you understand what "the model is learning" concretely means for a huge fraction of modern AI.

Start with the problem. A neural network's final layer spits out a list of raw numbers, one per possible answer, called logits. For a language model choosing the next token, that might be tens of thousands of numbers. These scores are unbounded and unnormalized: one could be 7.2, another -3.1, another 0.4. They are not probabilities, and you cannot compare them across examples or use them to say how confident the model is. You need a way to convert scores into a proper probability distribution.

That is exactly what softmax does. It exponentiates each score (making everything positive and amplifying gaps between big and small scores), then divides by the total so the results sum to one. A score of 7.2 next to a score of 0.4 becomes a large probability next to a tiny one; two similar scores become two similar probabilities. Crucially, softmax preserves order, the largest logit always becomes the largest probability, but it also encodes how much larger. Think of it as turning a set of loudness readings into "what fraction of the total noise each source made." This is the same operation covered from the decoding side in how AI picks its next word, where a temperature knob rescales the logits before softmax to make the model bolder or more cautious.

Now you have a predicted distribution, say 70% cat, 20% dog, 10% bird. How wrong is it? That is cross-entropy's job. Cross-entropy comes from information theory, and the intuition is about surprise: it measures how surprised the model is by the true answer. If the correct label is "cat" and the model assigned cat a high probability, the surprise is low and the loss is small. If the model confidently said "dog" and the answer was "cat," the surprise is enormous and the loss is large. Formally, for a single correct class, cross-entropy is just the negative logarithm of the probability the model gave to the right answer. Because of that logarithm, being confidently wrong is punished far more harshly than being uncertain, which pushes the model toward honest calibration rather than reckless guessing. This is the same quantity, averaged and exponentiated, that shows up as perplexity when people report how well a language model predicts text.

Why the pair matters so much comes down to their gradient, the signal that actually updates the weights during gradient descent via backpropagation. When you combine softmax with cross-entropy and work through the calculus, something beautiful happens: the gradient with respect to each logit simplifies to "predicted probability minus true label." If the model said 0.7 for the correct class, the gradient nudges that logit up in proportion to the 0.3 it was short; if it said 0.2 for a wrong class, the gradient pushes that logit down by 0.2. No messy terms, no exploding factors, just an error signal that is directly proportional to how far off the prediction was. That clean form is a large part of why this combination became the default across deep learning; it makes training stable and the learning signal easy to interpret. The canonical treatment is in Stanford's CS231n notes and Chapter 6 of the Deep Learning book by Goodfellow, Bengio, and Courville.

A few practical notes. In real implementations, softmax and cross-entropy are fused into one numerically stable operation (a "softmax cross-entropy" or "log-softmax plus negative log-likelihood" step) to avoid overflow when logits are large. The largest logit is subtracted from all of them first, which changes nothing mathematically but keeps the exponentials from blowing up. And for language models, the "classes" are the entire vocabulary, so this same tiny idea runs at every position of every sequence, billions of times during training.

The takeaway: softmax answers "how do I turn scores into a confident guess?" and cross-entropy answers "how wrong was that guess?" Almost every time a model is trained to choose among options, from labeling an image to predicting the next token, this is the target it is chasing, and the gradient it produces is the single cleanest reason the pair is everywhere.

Ground Truth. (2026, July 20). Softmax and Cross-Entropy: How a Model Turns Scores Into a Confident Guess. Ground Truth. https://groundtruth.day/learn/softmax-and-cross-entropy.html

@misc{groundtruth:softmax-and-cross-entropy,
  title  = {Softmax and Cross-Entropy: How a Model Turns Scores Into a Confident Guess},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/softmax-and-cross-entropy.html}
}

Topics: fundamentals · training · loss-functions · classification