CMU 11-785 L22 Revisiting EM algorithm and generative models

Key points

• EM: An iterative technique to estimate probability models for data with missing components or information
  • By iteratively “completing” the data and reestimating parameters
• PCA: Is actually a generative model for Gaussian data
  • Data lie close to a linear manifold, with orthogonal noise
  • A lienar autoencoder!
• Factor Analysis: Also a generative model for Gaussian data
• Data lie close to a linear manifold
• Like PCA, but without directional constraints on the noise (not necessarily orthogonal)

Generative models

Learning a generative model

• You are given some set of observed data X={ x}X=\{x\}X={ x}
• You choose a model $P(x;\theta )$ for the distribution of $x$
  • $\theta$ are the parameters of the model
• Estimate the theta such that $P(x;\theta )$ best “fits” the observations X={ x}X=\{x\}X={ x}
• How to define “best fits”?
  • Maximum likelihood!
  • Assumption: The data you have observed are very typical of the process

EM algorithm

• Tackle missing data and information problem in model estimation
• Let $o$ are observed data

$$\log P(o)=\log \sum _{h}P(h,o)=\log \sum _{h}Q(h)\frac{P(h,o)}{Q(h)}$$

• The logarithm is a concave function, therefore

log⁡∑hQ(h)P(h,o)Q(h)≥∑hQ(h)log⁡P(h,o)Q(h) \log \sum_{h} Q(h) \frac{P(h, o)}{Q(h)} \geq \sum_{h} Q(h) \log \frac{P(h, o)}{Q(h)} log