EM ALGORITHM

最新推荐文章于 2024-10-09 22:16:23 发布

原创最新推荐文章于 2024-10-09 22:16:23 发布 · 432 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#算法

Mathmatics 专栏收录该内容

2 篇文章

订阅专栏

Let $x$ be observable data and $z$ be latent data.

p ( x , z | θ ) p ( x | θ ) = p (z | x, θ)

$\frac{p\left(x,z|\theta \right)}{p\left(x|\theta \right)} = p\left(z|x,\theta \right)$

Take log on both sides:

\Rightarrow log p (x, z | θ) - log p (x | θ) = log p (z | x, θ) log p (x | θ) = log p (x, z | θ) - log p (z | x, θ)

$\begin{array}{rl} & \log p\left(x,z|\theta \right)-\log p\left(x|\theta \right) = \log p\left(z|x,\theta \right) \\ \Rightarrow & \log p \left(x|\theta \right) = \log p\left(x,z|\theta \right) - \log p\left(z|x,\theta \right) \end{array}$

Take conditional expectation with respect to $z|\theta ',x$ on both sides:

\Rightarrow ε [log p (x | θ) | θ', x] = ε [log p (x, z | θ) | θ', x] - ε [log p (z | x, θ) | θ', x] log p (x | θ) = ε [log p (x, z | θ) | θ', x] - ε [log p (z | x, θ) | θ', x]

$\begin{array}{rl} & \varepsilon \left[ \log p\left(x|\theta \right)|\theta ',x \right] = \varepsilon \left[ \log p\left(x,z|\theta \right) | \theta ',x\right] - \varepsilon \left[ \log p\left(z|x,\theta \right)| \theta ',x\right] \\ \Rightarrow & \log p \left(x|\theta \right) = \varepsilon \left[ \log p\left(x,z|\theta \right) | \theta ',x\right] - \varepsilon \left[ \log p\left(z|x,\theta \right)| \theta ',x\right] \end{array}$

Choose

\Rightarrow θ (i + 1) = arg max θ ε [log p (x, z | θ) | θ (i), x] θ (i + 1) = arg max θ \sum z p (z | θ (i), x) log p (x, z | θ)

$\begin{array}{rl} & \theta ^{\left(i+1\right)} = \arg \max \limits _{\theta} \varepsilon \left[ \log p\left(x,z|\theta \right) | \theta ^{\left(i\right)},x\right] \\ \Rightarrow & \theta ^{\left(i+1\right)} = \arg \max \limits _{\theta} \sum \limits _z p\left(z|\theta ^{\left(i\right)},x \right) \log p\left(x,z|\theta\right) \end{array}$

Prove that $p\left(x|\theta ^{\left(i\right)}\right)$ is increasing as $i$ increasing, i.e., $p\left(x|\theta ^{\left(i+1\right)} \right) \geq p\left(x|\theta ^{\left(i\right)}\right)$ .

Because of the choice of $\theta ^{\left(i+1\right)}$ , we have

ε [log p (x, z | θ (i + 1)) | θ (i), x] \geq ε [log p (x, z | θ (i)) | θ (i), x]

$\varepsilon \left[ \log p\left(x,z|\theta ^{\left(i+1\right)} \right) | \theta ^{\left(i\right)},x\right] \geq \varepsilon \left[ \log p\left(x,z|\theta ^{\left(i\right)} \right) | \theta ^{\left(i\right)},x\right]$

We only need to show that

ε [log p (z | x, θ (i + 1)) | θ (i), x] \leq ε [log p (z | x, θ (i + 1)) | θ (i), x]

$\varepsilon \left[ \log p\left(z|x,\theta ^{\left(i+1\right)} \right)| \theta ^{\left(i\right)},x\right] \leq \varepsilon \left[ \log p\left(z|x,\theta ^{\left(i+1\right)} \right)| \theta ^{\left(i\right)},x\right]$

This is true because of following.

If $\varepsilon$ is taken with respect to $p\left(x\right)$ , we have $\varepsilon \left[ \log p \left(x\right) \right] \geq \varepsilon \log p' \left(x\right)$ , where $p'\left(x\right)$ is any pdf (not identical as $p\left(x\right)$ ).

p.f.

\Rightarrow \Rightarrow \Rightarrow ε [log p ' ( x ) p ( x )] \leq log ε [p ' ( x ) p ( x )] (by Jensen's inequality) ε [log p' (x)] - ε [log p (x)] \leq log \int p ' ( x ) p ( x ) \cdot p (x) d x = 1 ε [log p' (x)] - ε [log p (x)] \leq 0 ε [log p' (x)] \leq ε [log p (x)]

$\begin{array}{rl} & \varepsilon \left[ \log \frac{p'\left(x\right)}{p\left(x\right)} \right] \leq \log \varepsilon \left[ \frac{p'\left(x\right)}{p\left(x\right)} \right] \qquad \left(\text{by Jensen's inequality}\right) \\ \Rightarrow & \varepsilon \left[ \log p'\left(x\right) \right] - \varepsilon \left[ \log p\left(x\right) \right] \leq \log \int \frac{p'\left(x\right)}{p\left(x\right)} \cdot p\left(x\right) dx = 1 \\ \Rightarrow & \varepsilon \left[ \log p'\left(x\right) \right] - \varepsilon \left[ \log p\left(x\right) \right] \leq 0 \\ \Rightarrow & \varepsilon \left[ \log p'\left(x\right) \right] \leq \varepsilon \left[ \log p\left(x\right) \right] \end{array}$

p.s.
Jensen’s inequality:

For a convex function $\phi$ ,