【the EM algorithm】期望最大化

最新推荐文章于 2018-12-26 21:26:56 发布

原创最新推荐文章于 2018-12-26 21:26:56 发布 · 585 阅读

0 ·

CC 4.0 BY-SA版权

Machine Learning 专栏收录该内容

3 篇文章

订阅专栏

本文深入探讨了EM算法的基本原理，从观察变量X出发，通过极大化对数似然函数，逐步推导出EM算法的数学表达形式。并给出了EM算法的具体步骤，包括E步和M步的迭代过程。

1. 一些英文简称

iid :independently and identically distribute

EM :Expectation Maximization

2. EM算法

首先来明确一下我们的目标：我们的目标是在观察变量X和给定观察样本 $\begin{Bmatrix} x_{1},x_{2},...,x_{n} \end{Bmatrix}$ 的情况下，极大化对数似然函数：

$l(\theta )=\sum_{i=1}^{n}lnP(X_{i}=x_{i}))$

其中只包含观察变量(Observed Variable)的概率密度函数

$P(X_{i}=x_{i})=\sum_{j}P(X_{i}=x_{i},Z=z_{i})$

其中Z为隐藏随机变量， $\begin{Bmatrix} z_{j} \end{Bmatrix}$ 是Z的所有可能取值。那么，

$l(\theta )=\sum_{i=1}^{n}ln\sum_{j}P(X_{i}=x_{i},Z=z_{j}) \=\sum_{i=1}^{n}ln\sum_{j}\alpha _{j}\frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}}$

$=\sum_{i=1}^{n}ln \sum_{j}\alpha _{j}\frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}}$

$\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}}$

其中， $\forall j,\alpha _{j}\in (0,1];\sum_{j}\alpha _{j}=1$ 。

上列不等式是有Jensen不等式得来的。它取等号的条件是：需要让随机变量变成常数值，这里得到：

$\frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}}=c \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}}$

那么：

$\alpha _{j}=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})} \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$={P(Z=z_{j}|X_{i}=x_{i})} \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

那么一般的EM算法的步骤如下：

$Repeat\: until\: convergence\: \{$

$E: for\:each\:i, \:calculate \=\alpha _{j}(z_{j})={P(Z=z_{j}|X_{i}=x_{i})} \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$Q_{i}(z^{(i)}):=p(z^{(i)}|x^{(i)};\theta ) \=\alpha _{j}(z_{j})={P(Z=z_{j}|X_{i}=x_{i})} \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$M: \=\alpha _{j}(z_{j})={P(Z=z_{j}|X_{i}=x_{i})} \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$\theta:=\textup{arg} \:\underset{\theta}{\textup{max} }\sum_{i}\sum_{z^{(i)}}Q_{i}(z^{(i)})\textup{log}\frac{p(x^{(i)},z^{(i)};\theta )}{Q_{i}(z^{(i)})} \=\alpha _{j}(z_{j})={P(Z=z_{j}|X_{i}=x_{i})} \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$\}$

期望最大化算法收敛性：

$l(\theta ^{(t+1)})=\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t+1)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j+1)}}{\alpha _{j}^{(j+1)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j+1)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$

$=l(\theta ^{(t)}) \=\geq\sum_{i=1}^{n}\sum_{j}\alpha _{j}^{(t)}ln \frac{P(X_{i}=x_{i},Z=z_{j})^{(j)}}{\alpha _{j}^{(j)}} \=\geq \sum_{i=1}^{n}\sum_{j}\alpha _{j}ln \frac{P(X_{i}=x_{i},Z=z_{j})}{\alpha _{j}} \=\frac{P(X_{i}=x_{i},Z=z_{j})}{P(X_{i}=x_{i})}=\alpha _{j}$