notes on entorpy 【笔记】熵

Shannon entropy 香农熵

The key concept of classical information theory is the Shannon entropy. Suppose we learn the value of a random variable X. The Shannon entropy of X quantifies how much information we gain, on average, when we learn the value of X. An alternative view is that the entropy of X measures the amount of uncertainty about X before we learn its value. These two views are complementary; we can view the entropy either as a measure of our uncertainty before we learn the value of X, or as a measure of how much information we have gained after we learn the value of X.

We often write the entropy as a function of a probability distribution, $p_1, \cdots, p_n$. The Shannon entropy associated with this probability distribution is defined by

$$H(X) \equiv H(p_1, \cdots, p_n) \equiv -\sum_{x} p_xlogp_x$$

The relative entropy 相对熵

The relative entropy is a very useful entropy-live measure of the closeness of two probability distributions, $p(x)$ and $q(x)$, over the same index set, $x$. Suppose $p(x)$ and $q(x)$ are two probability distributions on the same index set, x. Define the relative entropy of $p(x)$ to $q(x)$ by

$$H(p(x)||q(x)) \equiv \sum_x p(x)log\frac{p(x)}{q(x)} \equiv -H(p(x)) - \sum_xp(x)logq(x)$$

The cross entropy 交叉熵

$$CEH(p(x), q(x)) = E_p[-logq] = -\sum_x p(x)logq(x)$$

Reference

[1] Nielsen M A, Chuang I L. Quantum computation and quantum information[M]. Cambridge university press, 2010.
[2] http://blog.youkuaiyun.com/rtygbwwwerr/article/details/50778098