CRF、MRF、DBN的一些联系

CRF and MRF are two popular graphical models, which is an intuitive model to model all kinds of networks.

Recently deep belief network has aroused a huge surge in Machine Learning community, and it got state of the art results in numerous fields, like object recognition, acoustic understanding and topic modeling. DBN is also one kind of graphical models.

As we know, CRF is a discriminative model, while MRF is a generative model. Naive Bayes is probably the simplest MRF, while logistic regression is one kind of simple CRFs. The key difference between these two kinds of models is that: MRF is trying to model a joint distribution p(X,Y), however, CRF aims to build a conditional distribution p(Y|X). To be more clear, there are no kinds of potential functions like g(x1,x2) in CRF.  Nevertheless, the learning methods & procedures are quite the same, and maximum likelihood works for both model. By the way, for parameter learning in DBN, Prof. Hinton developed a method called contrastive divergence, which is pseudo-likelihood.

One important theorem in undirected graph is: 

Hammersley-Clifford theorem: a positive distribution p(y) > 0 satisfies the conditional independence properties of an undirected graph G if and only if p can be represented as a product of factors, one per maximal clique.