Study notes for Expectation Maximum Algorithm

最新推荐文章于 2024-10-27 06:14:16 发布

转载最新推荐文章于 2024-10-27 06:14:16 发布 · 77 阅读

本文介绍了EM算法的基本原理及其在存在缺失数据情况下的应用。通过凸函数与詹森不等式的概念，阐述了EM算法中期望步骤(E-step)与最大化步骤(M-step)的工作流程，并通过一个具体的例子展示了算法迭代过程直至收敛。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

The EM algorithm is an efficient iterative procedure to compute the maximum likelihood (ML) estimate in the presence of missing or hidden data (variables).
It intends to estimate the model parameters such that the observed data are the most likely.

Let $f$ be a real function defined on an interval $I=[a, b]$ . $f$ is said to be convex on $I$ if $\forall x_1, x_2\in I, \lambda\in [0,1]$ ,
$f(\lambda x_1+(1-\lambda)x_2) \le \lambda f(x_1)+(1-\lambda)f(x_2)$
$f$ is said to be strictly convex if the inequality is strict. Intuitively, this definition states that the function falls below (strictly convex) or is never above (convex) the straight line from points $(x_1, f(x_1))$ to $(x_2, f(x_2))$ .
$f$ is concave (strictly concave) if $-f$ is convex (strictly convex).
Theorem 1. If is twice differentiable on [a, b] and on [a, b], then is convex on [a, b].
- If x takes vector values, f(x) is convex if the hessian matrix H is positive semi-definite (H>=0).
- -ln(x) is strictly convex in (0, inf), and hence ln(x) is strictly concave in (0, inf).

The convexity is generalized to multivariate.
Let $f$ be a convex function defined on an interval $I$ . If $x_1, x_2, \ldots, x_n \in I$ and $\lambda_1, \lambda_2, \ldots, \lambda_n \ge 0$ with $\sum\nolimits_{i=1}^n \lambda_i=1$ ,
$f(\sum_{i=1}^n \lambda_ix_i)\le \sum_{i=1}^n \lambda_i f(x_i) \rightarrow f(E(x))\le E[f(x)]$
Note that $E[f(x)]=f(E(x))$ holds true if and only if $x=E(x)$ with probability 1, i.e., if X is a constant.
Hence, for concave functions:
$f(\sum_{i=1}^n \lambda_ix_i)\ge \sum_{i=1}^n \lambda_i f(x_i)$
Applying ln(x) and concavity, we can verify that,
$\frac{1}{n}\sum_{i=1}^n x_i \ge \sqrt[n]{x_1x_2...x_n}$

Each iteration of the EM algorithm consists of two processes
- E-step. The missing data are estimated, given the observed data and current estimate of the model parameters. The assumption is that the observed data is resulted from the parameters of a model.
- M-step. The likelihood function is maximized under the assumption that the missing data are known. That is, the estimate of the missing data from the E-step are used in lieu of the actual missing data. The typical likelihood function is the log likelihood function defined as $L(\theta)=ln P(X|\theta)$ , i.e. a function of the parameter theta given the observed data X.
Convergence is assured since the algorithm is guaranteed to increase the likelihood at each iteration.
The detailed derivation can be referred to Andrew's or Sean's tutorial.