Exponential family form of Multivariate Gaussian Distribution

最新推荐文章于 2020-09-13 21:31:43 发布

空言葉

最新推荐文章于 2020-09-13 21:31:43 发布

阅读量350

点赞数

CC 4.0 BY-SA版权

分类专栏：数据学习文章标签：指数族多元高斯分布

本文链接：https://blog.youkuaiyun.com/kongyanye/article/details/78488091

数据学习专栏收录该内容

1 篇文章

订阅专栏

Question

Exponential Family. Verify that the multivariate Gaussian distribution can be casted in exponential family form and derive expressions for $\eta$ , $b(\eta)$ , $T(\eta)$ , $a(\eta)$ .

Solution

The exponential family form is

p (y; η) = b (y) exp [η T T (y) - a (η)]

$p(y;\eta)=b(y)\exp[\eta^TT(y)-a(\eta)]$
The multivariate Gaussian distribution takes the form

N (x | μ, Σ) = 1 ( 2 π ) D / 2 | Σ | 1 / 2 exp [- 1 2 (x - μ) T Σ - 1 (x - μ)]

$\begin{equation} N(x|\mu, \Sigma)=\frac{1}{(2\pi)^{D/2}|\Sigma|^{1/2}}\exp[-\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)] \end{equation}$
Since

- 1 2 (x - μ) T Σ - 1 (x - μ) = - 1 2 (x T - μ T) Σ - 1 (x - μ) = - 1 2 (x T Σ - 1 x - x T Σ - 1 μ - μ T Σ - 1 x + μ T Σ - 1 μ) = - 1 2 (x T Σ - 1 x - 2 μ T Σ - 1 x + μ T Σ - 1 μ) = - 1 2 (v e c (Σ - 1) v e c (x x T) - 2 μ T Σ - 1 x + μ T Σ - 1 μ) = - 1 2 ([v e c (Σ - 1) - 2 μ T Σ - 1] [v e c (x x T) x] + μ T Σ - 1 μ) = [- 1 2 v e c (Σ - 1) μ T Σ - 1] [v e c (x x T) x] - 1 2 μ T Σ - 1 μ

$\begin{equation} \begin{aligned} -\frac{1}{2}(x-\mu)^T\Sigma^{-1}(x-\mu)&=-\frac{1}{2}(x^T-\mu^T)\Sigma^{-1}(x-\mu)\\ &=-\frac{1}{2}(x^T\Sigma^{-1}x-x^T\Sigma^{-1}\mu-\mu^T\Sigma^{-1}x+\mu^T\Sigma^{-1}\mu)\\ &=-\frac{1}{2}(x^T\Sigma^{-1}x-2\mu^T\Sigma^{-1}x+\mu^T\Sigma^{-1}\mu)\\ &=-\frac{1}{2}(vec(\Sigma^{-1})vec(xx^T)-2\mu^T\Sigma^{-1}x+\mu^T\Sigma^{-1}\mu)\\ &=-\frac{1}{2}(\left[\begin{array}{cc}vec(\Sigma^{-1})&-2\mu^T\Sigma^{-1}\end{array}\right] \left[\begin{array}{c} vec(xx^T)\\x\end{array}\right]+\mu^T\Sigma^{-1}\mu )\\ &=\left[\begin{array}{cc}-\frac{1}{2}vec(\Sigma^{-1})&\mu^T\Sigma^{-1}\end{array}\right] \left[\begin{array}{c} vec(xx^T)\\x\end{array}\right]-\frac{1}{2}\mu^T\Sigma^{-1}\mu \end{aligned} \end{equation}$
The 3rd equation is due to
(

Trace $Trace$ of a real number is still a real number and

Σ−1 $\Sigma^{-1}$ is a symmetric matrix)

x T Σ - 1 μ = T r (x T Σ - 1 μ) = T r (x T Σ - 1 μ) T = T r (μ T Σ - T x) = T r (μ T Σ - 1 x) = μ T Σ - 1 x

$\begin{equation} \begin{aligned} x^T\Sigma^{-1}\mu&=Tr(x^T\Sigma^{-1}\mu)=Tr(x^T\Sigma^{-1}\mu)^T\\&=Tr(\mu^T\Sigma^{-T}x)=Tr(\mu^T\Sigma^{-1}x)\\ &=\mu^T\Sigma^{-1}x \end{aligned} \end{equation}$
The 4th equation is due to

xTΣ−1x=vec(Σ−1)Tvec(xxT) $x^T\Sigma^{-1}x=vec(\Sigma^{-1})^Tvec(xx^T)$ , where

vec $vec$ means stack all columns of a matrix into one single column, i.e. if

x=⎡⎣⎢|x1||x2||…||xm|⎤⎦⎥ $x=\left[\begin{array}{cccc}|&|&|&|\\x_1&x_2&\ldots&x_m\\|&|&|&|\end{array}\right]$ , then

v e c (x) = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ | x 1 | x 2 | ⋮ | x m | ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥

$vec(x)=\left[\begin{array}{c}|\\x_1\\|\\x_2\\|\\\vdots\\|\\x_m\\|\end{array}\right]$
So we can derive the parameter of exponential family as below:

b (y) = 1 ( 2 π ) D / 2 η = ⎡ ⎣ - 1 2 v e c (Σ - 1) μ T Σ - 1 ⎤ ⎦ T (y) = [v e c (x x T) x] a (η) = 1 2 μ T Σ - 1 μ - ln | Σ | 1 / 2

$\begin{equation} \begin{aligned} &b(y)=\frac{1}{(2\pi)^{D/2}}\\ &\eta=\left[\begin{array}{c}-\frac{1}{2}vec(\Sigma^{-1})\\\mu^T\Sigma^{-1}\end{array}\right]\\ &T(y)=\left[\begin{array}{c} vec(xx^T)\\x\end{array}\right]\\ &a(\eta)=\frac{1}{2}\mu^T\Sigma^{-1}\mu-\ln|\Sigma|^{1/2} \end{aligned} \end{equation}$