[Elements of Information Theory]

Entropy

Definition

• Let X be a discrete random variable with alphabet $\mathcal{X}$ and probability mass function $p(x)=Pr(X=x),xϵ\mathcal{X}.$

• The entropy of X is defined as
  $$H(X)=-\sum _{xϵ\mathcal{X}}p(x)logp(x)$$
  a measure of a uncertainty of a random variable

• $H(X)$ only depends on $p(x)$.We also write $H(p)$ for $H(X)$.

• $H(X)\ge 0$

• When $X$ is uniform over $\mathcal{X}$,then $H(X)=log|\mathcal{X}|$

• $H_b(X)=lo{g}_{b}a\ast H_a(X)$

Example

• Binary entropy function $H(p)$
  $$LetX=\{\begin{array}{ll}1& \text{with probability p}\\ 0& \text{with probability 1-p}\end{array}$$
  $$H(X)=-plog(p)-(1-p)log(1-p)$$

• $H(X)=-E_p[logp(X)]$

• For a discrete random variable $X$ defined on $\mathcal{X}$,
  $$0\le H(X)\le log|\mathcal{X}|$$
  Equality if and only if $p(x)=1/|\mathcal{X}|$.(Uniform distribution maximizes entropy)

• Convexity is widely applied
  $$\sum _i{p}_{i}f(x_i)\le f(\sum _i{p}_i{x}_i)$$

Joint Entropy

• Two random variables $X$ and $Y$ can be considered to be a single vector-valued random variable
• The joint entropy $H(X,Y)$ of a pair of discrete random variable $(X,Y)$ with joint distribution $p(x,y)$ is defined as
  $$H(X,Y)=-\sum _{xϵ\mathcal{X}}\sum _{yϵ\mathcal{Y}}p(x,y)logp(x,y)$$
• $H(X,Y)=-Elogp(X,Y)$
• $H(X,X)=H(X)$
• $H(X,Y)=H(Y,X)$
• $H(X_1,X_2,...,X_n)=-\sum p(x_1,x_2,...,x_n)logp(x_1,x_2,...,x_n)$

Conditional Entropy

• Entropy for $p(Y|X=x)$
  $$H(Y|X=x)=\sum _y-p(y|X=x)logp(y|X=x)=-Elogp(y|X=x)$$
• When $X$ is known:$H(Y|X)\le H(Y)$
• $H(X|Y)\ne H(Y|X)$
• $H(X|Y)+H(Y)=H(Y|X)+H(X)=H(X,Y)$

Zero Entropy

• If $H(Y|X)=0$:
  • then $Y$ is a function of $X$.
  • $H(Y|X=x)=0$