What Are Softmax and Cross-Entropy? Turning Logits into Probabilities and Training Loss

Softmax and cross-entropy are a combination that lets a multiclass neural network express which class is correct among several candidates as probabilities and numerically evaluate how different that prediction is from the correct answer. Softmax converts the model's raw outputs, called logits, into a probability distribution that sums to 1. Cross-entropy calculates the difference between that probability distribution and the target distribution as a loss. Training is the process of adjusting the model's weights to reduce this loss.

For example, suppose you are classifying an image as one of cat, dog, or bird. The model often does not initially respond with something like “70% probability of cat.” Instead, it produces logits: comparative scores for the three classes. Softmax normalizes these scores into comparable probabilities, and cross-entropy assigns a penalty according to how much probability the model assigned to the actual answer. Understanding both together can reduce common confusion about the final output layer, label formats, and loss-function configuration.

What does the softmax function do?

Softmax is a function that converts a vector of logits $z=(z_1,\dots,z_K)$ for $K$ classes into class probabilities $p=(p_1,\dots,p_K)$. The probability of class $i$ is calculated as follows.

Here, the exponential function $e^{z_i}$ gives classes with larger scores greater relative weight. Because the denominator is the sum of the exponentiated scores for all classes, every output falls between 0 and 1, and the total probability sums to 1. Thus, softmax output represents the relative likelihood that an input belongs to each class as a single probability distribution. PyTorch documentation likewise describes softmax as rescaling each input element to the range from 0 to 1 so that the elements sum to 1. docs.pytorch.org↗

Softmax is particularly appropriate when the candidates are mutually exclusive. That is, if one is true, the others cannot be the correct answer. Examples include classifying an image whose main animal is exactly one of cat, dog, or bird, or assigning a sentence to exactly one predefined sentiment category.

In contrast, softmax probabilities are not independent of one another. If the probability of cat rises, the probability of dog or bird must decrease relatively to keep the three probabilities summing to 1. This characteristic is usually not appropriate for cases where multiple labels can be true simultaneously, such as a problem in which both a person and a bicycle can appear in one image. In that case, consider applying sigmoid independently to each class.

How are logits different from probabilities?

A logit is an unnormalized score produced by a final linear layer or similar component. Logits can be negative or greater than 1, and the sum of logits across classes has no particular meaning. For example, $[2,1,0]$ can be a set of logits for three classes, but it is not a probability vector.

An important characteristic of logits is that the differences between classes matter more than their absolute values. Adding the same constant $c$ to every logit does not change the softmax result.

This is because both the numerator and denominator are multiplied by the common factor $e^c$, which cancels out. Therefore, logits $[2,1,0]$ and $[102,101,100]$ may appear to be different raw scores, but they produce the same softmax probabilities. Softmax turns relative differences among candidates, rather than the scores' reference point, into probabilities.

What does cross-entropy measure?

Cross-entropy is a loss that indicates how different the actual target distribution $y$ is from the distribution $p$ predicted by the model. Its basic multiclass form is as follows.

Here, $y_i$ is the weight that the actual label assigns to class $i$, and $p_i$ is the probability predicted by the model for class $i$. A smaller loss means that the predicted distribution is closer to the target distribution. When the natural logarithm is used, the unit is the nat.

This formula is not simply a calculation that separates correct and incorrect answers as 0 or 1. It reflects not only whether the model got the answer right, but also how confident it was that the answer was correct. A prediction assigning 0.51 to the correct class and one assigning 0.99 can have the same accuracy if the highest-probability class is correct in both cases. However, cross-entropy gives the latter a lower loss. This is why training encourages the model not merely to rank the correct answer first, but to assign it a higher probability.

Why does the formula simplify for one-hot labels?

Targets for mutually exclusive classification are often represented as one-hot vectors. If the first class is correct among three classes, the label is $[1,0,0]$. Because only the target class $t$ has $y_t=1$ and the rest are 0, cross-entropy reduces to the following.

In other words, the loss is determined only by the predicted probability $p_t$ of the correct class. As $p_t$ approaches 1, $-\log p_t$ approaches 0. In contrast, as $p_t$ becomes smaller, the loss grows quickly. Because of this property, predictions that assign a low probability to the correct answer—especially those that predict an incorrect answer with very high confidence—receive a large penalty during training.

Theoretically, if the correct-class probability is exactly 0, $\log 0$ is undefined and the loss diverges to infinity. Actual neural-network implementations usually use stable calculations based on logits, avoiding the approach of explicitly producing a probability of 0 and then taking its logarithm.

How do softmax and cross-entropy connect?

The final part of a typical multiclass model can be understood in three steps.

1. The final linear layer outputs class logits $z$.
2. Softmax converts the logits into probabilities $p$.
3. Cross-entropy compares probabilities $p$ with the target label $y$ to calculate the loss $L$.

For a one-hot target, this connection can be expanded as follows.

This expression is often called softmax cross-entropy or categorical cross-entropy. The first term, $-z_t$, becomes smaller as the correct-class logit increases, while the second term, $\log\sum_j e^{z_j}$, also considers the scores of every competing class. The model therefore learns to increase the score of the correct class while making it relatively more prominent than incorrect classes.

Here, softmax and cross-entropy should be distinguished conceptually. Softmax is a function that changes how outputs are represented, while cross-entropy is a loss that quantifies the training objective. In practical frameworks, however, the two steps are often provided as one combined loss function, which can make them seem like a single function.

What does the calculation reveal with numbers?

Consider a three-class problem with logits $[2,1,0]$. Applying softmax gives approximate probabilities of $[0.665,0.245,0.090]$. TensorFlow's softmax example presents the same approximate probabilities for these logits. docs.pytorch.org↗

If the first class is correct, the one-hot label is $[1,0,0]$, and the loss is as follows.

The first class has the highest probability among the three candidates, and that probability is about 0.665, so the loss is relatively small. However, “small” does not indicate an absolute passing threshold. An appropriate loss magnitude can vary with the number of classes, the difficulty of the data, label uncertainty, and whether loss is summed or averaged across a batch.

If, for the same logits, the third class were actually correct, the loss would be $-\log(0.090)$ and therefore much larger. The model viewed the third class as the least likely candidate. In this way, cross-entropy sensitively reflects not only whether a prediction is wrong, but also how confidently it is wrong.

Why should probability ranking and loss value be considered together?

Accuracy usually selects the single class with the largest probability and counts whether it matches the correct answer. Therefore, a prediction of $[0.34,0.33,0.33]$ when the first class is correct and a prediction of $[0.99,0.005,0.005]$ when the first class is correct are both counted as one correct prediction.

Cross-entropy does not treat these equally. The first prediction barely distinguishes among the three classes, while the second assigns a very high probability to the correct answer. Conversely, when the highest-probability class is incorrect, the loss also varies according to how low the correct-class probability was. For this reason, training generally minimizes cross-entropy, while evaluation also checks separate metrics such as accuracy according to the task objective.

Why should logits be passed directly to the training loss function?

Many deep-learning frameworks' combined loss functions take logits, not softmax probabilities, as input. For example, TensorFlow's softmax_cross_entropy_with_logits, as its name indicates, accepts logits and internally performs the calculations corresponding to softmax and cross-entropy. Its documentation warns against supplying outputs to which softmax has already been applied. www.tensorflow.org↗

If you violate this rule by applying softmax at the end of the model and then passing the result to a loss function intended for logits, the probabilities may be interpreted as logits again rather than receiving the intended single softmax application. This changes the meaning of the loss and its gradients, causing the training configuration to be incorrect. Whether a loss “includes softmax internally” or “accepts already converted probabilities” should be verified from the API's input contract, not from the function name alone.

The purpose of inference or display is different. After training, you can apply softmax to logits when you want to show class probabilities or perform probability-based post-processing. The key is not that softmax should never be used, but that it should be applied exactly once at the input stage expected by the training loss.

Why is numerical stability important?

The softmax formula contains exponentials and logarithms. If logits are extremely large or small, calculating $e^{z_i}$ can exceed a computer's representable range, creating a risk of overflow or underflow. Calculating probabilities first and then applying logarithms separately can be vulnerable to these problems.

Combined loss functions and log_softmax-family implementations generally use stable ways to calculate $\log\sum_j e^{z_j}$. PyTorch documentation also explains that log_softmax has better numerical properties than calculating softmax and logarithm separately. docs.pytorch.org↗

The constant-shift property described earlier can also be used for stabilization. Subtracting the maximum value $m=\max_j z_j$ from every logit does not change the softmax result.

This makes the largest exponential term $e^0=1$, reducing the need to handle unnecessarily large numbers. In production code, however, it is generally preferable to use a framework-provided combined loss based on logits or a stable log-softmax operation rather than implementing the components separately yourself.

Should labels be one-hot vectors or integers?

Label representation is connected to the input contract of the loss function. Common formats include one-hot or probability labels, and sparse integer labels.

One-hot labels correspond directly to the basic cross-entropy formula. Sparse integer labels can store the same information more compactly, which can be useful when there are many classes. However, the two formats have different tensor shapes and data types, so you must use the format required by a particular loss function.

TensorFlow's documentation for logit-based softmax cross-entropy explains that label vectors must be valid probability distributions and distinguishes related label formats for mutually exclusive classification. www.tensorflow.org↗ For example, passing a single integer directly to a one-hot loss function, or passing a one-hot vector to a sparse integer-label function, may not calculate the intended quantity.

Probability labels relax the assumption that exactly one class must have a value of 1. However, this does not mean the same thing as a multilabel problem where several classes can independently be true. In softmax classification, probability labels still represent one distribution whose elements sum to 1.

How do binary classification, multiclass classification, and multilabel classification differ?

Output functions and losses should not be chosen only by looking at the number of classes. First determine whether target events can occur at the same time. The following distinction is useful in practice.

In multiclass classification, it is natural for class probabilities to sum to 1. If the probability that an image is a cat rises, the probability that the same image is a dog or bird should decrease relatively. TensorFlow documentation also distinguishes the use of softmax cross-entropy when labels are mutually exclusive from situations with independent multiple labels, which require a different setup. www.tensorflow.org↗

In multilabel classification, each output is closer to a separate question: “Is there a person?”, “Is there a bicycle?”, and “Is there a tree?” can all be answered yes simultaneously. If softmax forces the total to 1, increasing the probability of one label forces the probabilities of other labels down, which may not fit the problem structure. Applying sigmoid to each class allows the probability of each item to be handled independently.

Binary classification can be especially confusing because both approaches appear possible. Applying sigmoid to one output directly represents the probability of the positive class. Applying softmax to two logits represents positive and negative as a two-class distribution summing to 1. Whichever approach you choose, the label encoding and loss function must match that output format.

Does minimizing cross-entropy mean matching distributions?

When both the target $y$ and prediction $p$ are valid probability distributions, cross-entropy can be expressed through its relationship to entropy and KL divergence as follows.

Here, $H(y)$ is the entropy representing the uncertainty of the true distribution itself, and $D_{\mathrm{KL}}(y\Vert p)$ is the KL divergence representing the difference between the true distribution $y$ and predicted distribution $p$. If the target distribution $y$ in the training data is fixed, $H(y)$ does not change with the model parameters. Therefore, aside from that constant, reducing cross-entropy can be interpreted as reducing the KL divergence between $y$ and $p$.

This interpretation shows why cross-entropy is widely used as a learning objective for probability distributions. The model is pressured not only to place the correct class at the top, but also to distribute probabilities close to the observed target distribution.

However, this equation has assumptions. Both $y$ and $p$ must be probability distributions: their components must be in the appropriate range and sum to 1. Logits themselves are not probability distributions, so they are not directly substituted into this relationship. Logits connect to probability-distribution calculations through softmax or a combined loss based on logits.

What are common misconceptions and errors?

First is the misconception that logits are probabilities. The class with the largest logit also has the largest softmax probability, but logit values themselves cannot be read as percentages. How much a logit difference such as 2 versus 1 translates into a probability difference is determined only after applying softmax together with all other logits.

Second is the misconception that classification training works only if softmax is added to the end of the model. If you use a combined cross-entropy loss that accepts logits, it is common not to explicitly add final softmax during training. This supports the internal calculation and numerical stability. www.tensorflow.org↗docs.pytorch.org↗

Third is the misconception that softmax cross-entropy suits every classification problem. This loss combination is natural when classes are mutually exclusive. If multiple correct tags can coexist for one sample, softmax, which constrains total class probability to 1, may fail to represent the meaning of the data properly.

Fourth is the misconception that cross-entropy and accuracy are the same metric. Accuracy generally looks only at the single maximum-probability class, while cross-entropy reflects the full probability assigned to the correct answer. Therefore, two models with the same accuracy can have different cross-entropy values. This is also one reason training loss can decrease without accuracy rising monotonically at every step.

Fifth is the error of judging whether a model is good or bad from a single loss value alone. Cross-entropy values are affected by the number of classes, task difficulty, label format, and aggregation method. It is more appropriate to consider changes under the same task and configuration, loss on validation data, and separate evaluation metrics suitable for the task.

What should you check before configuring the model?

Before applying softmax and cross-entropy, it helps to check the following questions in order.

1. Does each sample have exactly one correct class? If so, softmax-based multiclass classification is a candidate. If multiple labels can be true simultaneously, first consider class-wise sigmoid and binary loss.
2. Is the model's final output logits or already probabilities? Logits are unconstrained raw scores, while probabilities are normalized values. If the combined loss expects logits, do not apply softmax in advance.
3. Does the label format match the loss function? Check whether it accepts one-hot or probability vectors, or integer indices. In addition to the format, check the batch dimension and the position of the class dimension.
4. Have you distinguished the purposes of training and inference? During training, calculate with a stable logit-based loss. During inference, use softmax when you need to display probabilities or apply probability-based rules.
5. Do you interpret loss and evaluation metrics separately? Cross-entropy is a training objective that reflects the quality of probability predictions, whereas accuracy measures whether the final choice matches the target. Interpret them together according to the task.

This checklist is not limited to a particular library. However, even when actual API names are similar, they can differ in whether they expect logits, probabilities, or integer labels as inputs. Immediately before implementation, check the official documentation for the function you use to confirm the input form and whether it includes an internal activation function.

Conclusion: Understand probability conversion and loss calculation by their roles

Softmax converts multiple logits into a probability distribution that sums to 1, representing the relative likelihood of mutually exclusive classes. Cross-entropy calculates the difference between that predicted distribution and the target distribution as a loss, assigning a larger penalty to predictions that assign a low probability to the correct answer. For a one-hot target, the loss simplifies to $-\log p_t$, making the learning objective of increasing the correct-class probability clear.

In practice, softmax and cross-entropy are often not calculated separately. Instead, a combined loss function that accepts logits is used. This preserves the mathematical objective while enabling more numerically stable computation. However, this combination is most natural when the problem structure is “multiclass classification with exactly one correct answer.” When selecting outputs and losses, the key is to check whether labels are mutually exclusive and what input the loss function expects before considering the number of classes.