loss functions for NN

cross entropy or log loss

In binary classification, the output is a single probability value $y$.

$$\mathcal{L}(x, z)=-(z\ln(y)+(1-z)\ln(1-y))$$

For multi-class problem, the output is a vector $\mathbf{y}$,

$$\mathcal{L}(x, \mathbf{z})=-\sum_{k=1}^{K}\mathbf{z}_k\ln \mathbf{y}_k$$

References

• Loss Function Semantics
• Loss Functions for Preference Levels
• Are Loss Functions All the Same?
• How do you decide which loss function to use for machine learning?
• Loss functions; a unifying view
• Softmax vs. Softmax-Loss: Numerical Stability
• Relationship Between Logistic Loss and Cross Entropy Loss
• Why You Should Use Cross-Entropy Error Instead Of Classification Error Or Mean Squared Error For Neural Network Classifier Training