entropy, crosss entropy

最新推荐文章于 2019-03-07 23:30:06 发布

原创最新推荐文章于 2019-03-07 23:30:06 发布 · 246 阅读

0 ·

CC 4.0 BY-SA版权

ai 专栏收录该内容

15 篇文章

订阅专栏

本文介绍了信息论中的两个核心概念：信息熵与交叉熵。信息熵衡量了接收前信息的不确定性，而交叉熵则用于量化两个概率分布之间的差异。文章通过实例解释了如何计算英语字符的信息熵，并探讨了在机器学习中交叉熵的应用。

Information Entropy

Information Entropy measures the information missing before reception, saying the level of uncertainty of a random variable $X$ .
Information entropy definition:

H (X) = - \sum i = 1 n p (x i) log p (x i)

$H(X)=-\sum_{i=1}^{n}p(x_i)\operatorname{log}p(x_i)$
where,

X X $X$ is a random variable,

p(xi) p ( x i ) $p(x_i)$ is the probability of

X=xi X = x i $X=x_i$ . When

log l o g $log$ is

log2 l o g 2 $log_2$ the unit of

H(X) H ( X ) $H(X)$ is bit. When

log l o g $log$ is

log10 l o g 10 $log_{10}$ the unit of

H(X) H ( X ) $H(X)$ is dit.

Example

English character

$X$ is a random variable. It could be one character of $a, b, c ... x, y, z$ . The information entropy of $X$ :

H (X) = - \sum i = 1 26 1 26 * l o g 2 1 26 = 4.7

$\begin{aligned} H(X)&=-\sum_{i=1}^{26}\frac{1}{26}*\operatorname{log_2}\frac{1}{26}\\ &=4.7\\ \end{aligned}$
This means the information entropy of a English character is 4.7 bit, meaning 5 binary numbers are able to encode a English character.

ASCII code

$X$ is a random variable. It could be one ASCII code. The total number of ASCII code is 128. The information entropy of $X$ :

H (X) = - \sum i = 1 128 1 128 * l o g 2 1 128 = 7

$\begin{aligned} H(X)&=-\sum_{i=1}^{128}\frac{1}{128}*log_2\frac{1}{128}\\ &=7\\ \end{aligned}$
This means the information entropy of a English character is 7 bit, meaning 7 binary numbers are able to encode an ASCII code. We use a Byte, which is 8 bits, to stand for a ASCII code. The extra one bit is used for checking.

================================================

Cross Entropy in Machine Learning

In information theory, the cross entropy between two probability distributions $p$ and $q$ over the same underlying set of events measures the average number of bits needed to identify an event drawn from the set, if a coding scheme is used that is optimized for an “unnatural” probability distribution $q$ , rather than the “true” distribution $p$ .
Cross entropy definition:

S (p, q) = - \sum x p (x) log q (x)

$S(p, q)=-\sum_{x}p(x)\operatorname{log}q(x)$
where,

x x $x$ is each certain value of the set.

p p $p$ is the target distribution,

q q $q$ is the temporary, unreal or unnatural distribution.

The more similar $p$ and $q$ , the smaller $S(p, q)$ . So $S(.)$ could be used as training target. There is an application, named tf.nn.softmax_cross_entropy_with_logits_v2(), in tensorflow for this.

Example

The training instance one-hot label is $y\_target=[0, 1, 0, 0, 0]$ . The one-hot label calculated by you algorithm $y\_tmp=[0.1, 0.1, 0.2, 0.1, 0.5]$ .

You want to make $y\_tmp$ approximating $y\_target$ . In other word, you goal is to make $y\_tmp[0]$ smaller, to make $y\_tmp[1]$ greater, to make $y\_tmp[2]$ smaller … It’s a complex task. so much to be consider.