CMU 11-785 L23 Variational Autoencoders

EM for PCA

With complete information

• If we knew $z$ for each $x$, estimating $A$ and $D$ would be simple

$$x=Az+E$$

$$P(x\mid z)=N(Az,D)$$

• Given complete information $\left(x_1,z_1\right),\left(x_2,z_2\right)$

$$\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x,z)=\underset{A,D}{\mathrm{argmax}}\sum _{(x,z)}\log P(x\mid z)$$

$$=\underset{A,D}{\mathrm{argmax}}\sum _{(x,Z)}\log \frac{1}{\sqrt{(2\pi {)}^d|D|}}\exp \left(-0.5(x-Az{)}^T{D}^{-1}(x-Az)\right)$$

• We can get a close form solution: $A=X{Z}^{+}$
• But we don’t have $Z$ => missing

With incomplete information

• Initialize the plane
• Complete the data by computing the appropriate $z$ for the plane
  • $P(z|X;A)$ is a delta, because $E$ is orthogonal to $A$
• Reestimate the plane using the $z$
• Iterate

Linear Gaussian Model

• PCA assumes the noise is always orthogonal to the data
  • Not always true
• The noise added to the output of the encoder can lie in any direction (uncorrelated)
• We want a generative model: to generate any point
  • Take a Gaussian step on the hyperplane
  • Add full-rank Gaussian uncorrelated noise that is independent of the position on the hyperplane
    • Uncorrelated: diagonal covariance matrix
    • Direction of noise is unconstrained

With complete information

x=Az+e x=A z+e