Kullback-Leibler (KL) loss

Kullback-Leibler($\mathrm{K}\mathrm{L}$) loss
（离散）For discrete probability distributions $F(x)$ and $G(x)$, the Kullback-Leibler ($\mathrm{K}\mathrm{L}$) loss from $F(x)$ to $G(x)$ is defined[5] to be
KL{F(x)∥G(x)}=∑i=1nF(x)log⁡F(x)G(x). \mathrm {KL}\{F(x)\|G(x)\} = \sum_{i=1}^nF(x)\log\frac{F(x)}{G(x)}.KL{F(x)∥G(x)}=i=1∑n​F(x)logG(x)F(x)​.
（连续）For distributions $F(x)$ and $G(x)$ of a continuous random variable, the Kullback–Leibler($\mathrm{K}\mathrm{L}$) loss is defined to be
KL{F(x)∥G(x)}=∫−∞∞f(x)log⁡f(x)g(x)dx\mathrm {KL}\{F(x)\|G(x)\} = \int_{-\infty}^{\infty}f(x)\log\frac{f(x)}{g(x)}dx KL{F(x)∥G(x)}=∫−∞∞​f(x)logg(x)f(x)​dx
where $f(x)$ and $g(x)$ is the densities function of $F(x)$ and $G(x)$.

The Kullback–Leibler loss is always non-negative(始终非负), that is
KL{F(x)∥G(x)}⩾0.\mathrm {KL}\{F(x)\|G(x)\}\geqslant0.KL{F(x)∥G(x)}⩾0.
The Kullback–Leibler($\mathrm{K}\mathrm{L}$) loss KL{F(x)∥G(x)}\mathrm {KL}\{F(x)\|G(x)\}KL{F(x)∥G(x)} is convex(凸的) in the pair of probability mass functions $(f,g)$, i.e. if $(f_1,g_1)$ and $(f_2,g_2)$ are two pairs of probability mass functions, then KL{λf1+(1−λ)f2∥λg1+(1−λ)g2}≤λKL(f1∥g1)+(1−λ)KL(f2∥g2){\mathrm {KL}\{\lambda f_{1}+(1-\lambda )f_{2}\|\lambda g_{1}+(1-\lambda )g_{2}\}\leq \lambda \mathrm {KL} (f_{1}\|g_{1})+(1-\lambda )\mathrm {KL} (f_{2}\|g_{2})}KL{λf1​+(1−λ)f2​∥λg1​+(1−λ)g2​}≤λKL(f1​∥g1​)+(1−λ)KL(f2​∥g2​) for $0\le \lambda \le 1$.

eg: Multivariate normal distributions
Suppose that we have two multivariate normal distributions, with means ${\mu }_0,{\mu }_1$ and with (nonsingular) covariance matrices ${\Sigma }_0,{\Sigma }_1$. If the two distributions have the same dimension, k, then the Kullback–Leibler($\mathrm{K}\mathrm{L}$ ) loss between the distributions is as follows:
$$\mathrm{K}\mathrm{L}({\mathcal{N}}_0\Vert {\mathcal{N}}_1)=\frac{1}{2}\left\{\mathrm{t}\mathrm{r}\left({\Sigma }_1^{-1}{\Sigma }_0\right)+{\left({\mu }_1-{\mu }_0\right)}^{\text{T}}{\Sigma }_1^{-1}({\mu }_1-{\mu }_0)-k+\log \left(\frac{\det {\Sigma }_1}{\det {\Sigma }_0}\right)\right\}.$$