EM算法和FA

EM

极大似然估计：要求样本来自同一个分布，然后估计参数。
EM：为个体指定一个参数–>估计参数–>调节分布–>估计参数。
E-step : repeatedly construct a low-bound on $l$.
M-step : optimize that low-bound.
details are as follows:
EM : repeat until convergence{
(E-step) for each i ,set

$$Q_i(z^{(i)}):=p(z^{(i)}|x^{(i)};\theta)$$
(M-step) set

$$\theta:={argmax}_\theta\sum_i\sum_{z^{(i)}}Q_i(z^{(i)})log\frac{p(x^{(i)},z^{(i)};\theta)}{Q_i(z^{(i)})}$$
compensatory acknowlege:
1) Jensen’s inequality
$f$ is a convex function , $x$ is a random variable , then $$E[f(x)]\ge f[E(x)]$$
2) 多元正态分布：若 $$x\sim N_p(\mu,\Sigma)$$ $x$服从$p$维正态分布，且$Ex = \mu~,~D(x) = \Sigma=E[(x-Ex)(x-Ex)^T]~,$则 $$f(x_1,x_2……x_p)=\frac1{{{2\pi}^\frac p2}|\Sigma|^\frac12}\exp^{-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)}$$ 如果，$y=cx+b~,c\in R^{M\times1}~,$则 $$Y\sim N_m(c\mu+b,c\Sigma c^T)$$

FA

A joint distribution on $(x,z)$ as follows :

$$z\sim N(0,I),x|z\sim N(\mu+\Lambda z,\psi)$$ $z\in R^k,\mu\in R^n,\Lambda\in R^{n\times k},\psi\in R^{n\times n}$,$\psi$ is a diagonal matrix and $k$ is usually chosen to be smaller than n .
Imagining that $x^{(i)}$ is generated by sampling a $k$ dimension multivariate gaussian $z^{(i)}$ $$x = \mu+\Lambda z +\epsilon, \begin {bmatrix}z\ x\end{bmatrix}^T\sim N(\mu_{zx},\Sigma)$$ $z\sim N(0,I),\epsilon\sim N(0,\psi),\mu_{zx} = \begin{bmatrix}0\\ \mu\end{bmatrix},\Sigma = \begin {bmatrix} I&\Lambda ^T\\ \Lambda & {\Lambda \Lambda ^T+\psi}\\ \end {bmatrix}$ so log likelihood of the parameters is as follows : $$l(\mu,\Lambda,\Psi)=log\Pi_{i = 1}^m\frac1{(2\pi)^{(\frac n2)}|\Lambda \Lambda^T+\psi|^\frac12}exp[-\frac12(x^{(i)}-\mu)^T(\Lambda \Lambda^T+\psi)^{-1}(x{(i)-\mu})]$$ then use EM algorithm .
编辑公式会有眼花缭乱的感觉~