PRML推导(1) - 高斯分布中的两个小问题

一元高斯分布的方差

考虑一元实值变量 $x$ , 其高斯分布有如下定义：
$$\mathcal{N}\left(x|\mu ,{\sigma }^2\right)=\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{\left(x-\mu \right)}^2\right\}\text{\hspace{1em}(1.1)}$$

对于变量 $x$，其方差为： $var[x]=\mathbb{E}[x^2]-\mathbb{E}[x{]}^2$，前一项即为 $x$ 的二阶原始矩：
$$\mathbb{E}\left[x^2\right]={\int }_{-\infty }^{\infty }{x}^2\mathcal{N}\left(x|\mu ,{\sigma }^2\right)dx={\int }_{-\infty }^{\infty }{x}^2\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{\left(x-\mu \right)}^2\right\}dx\text{\hspace{1em}(1.2)}$$

采用换元法，令 $t=x-\mu$ ,即：
$$\begin{array}{rl}\mathbb{E}\left[x^2\right]& ={\int }_{-\infty }^{\infty }{\left(t+\mu \right)}^2\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{t}^2\right\}d\left(t+\mu \right)\\ & ={\int }_{-\infty }^{\infty }\left(t^2+2t\mu +{\mu }^2\right)\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{t}^2\right\}dt\\ & ={\int }_{-\infty }^{\infty }{t}^2\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{t}^2\right\}dt+\mu {\int }_{-\infty }^{\infty }t\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{t}^2\right\}dt+{\mu }^2{\int }_{-\infty }^{\infty }\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{t}^2\right\}dt\end{array}(1.3)$$
其中第二项中 $t\mathcal{N}\left(t|0,{\sigma }^2\right)$ 为奇函数，即积分为0；第三项 $\mathcal{N}\left(t|0,{\sigma }^2\right)$积分为1，第三项值为 ${\mu }^2$；对于第一项，为了更直观的表示，依次采用两次换元法，令 $t=\sqrt{2}\sigma m$ 以及 $y=m^2$,则：
$$\begin{array}{rl}{\int }_{-\infty }^{\infty }{t}^2\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{t}^2\right\}dt& ={\int }_{-\infty }^{\infty }{\left(\sqrt{2}\sigma m\right)}^2\frac{1}{{\left(2\pi {\sigma }^2\right)}^{\frac{1}{2}}}\exp \left\{-\frac{1}{2{\sigma }^2}{\left(\sqrt{2}\sigma m\right)}^2\right\}d\left(\sqrt{2}\sigma m\right)\\ & ={\int }_{-\infty }^{\infty }\frac{2{\sigma }^2}{\sqrt{\pi }}{m}^2{e}^{-m^2}dm\\ & ={\int }_{-\infty }^{\infty }\frac{2{\sigma }^2}{\sqrt{\pi }}y{e}^{-y}d\sqrt{y}\\ & =2{\int }_0^{\infty }\begin{array}{c}\begin{array}{c}\frac{{\sigma }^2}{\sqrt{\pi }}{y}^{\frac{3}{2}-1}{e}^{-y}dy\end{array}\end{array}\\ & =\frac{2{\sigma }^2}{\sqrt{\pi }}\Gamma \left(\frac{3}{2}\right)=\frac{{\sigma }^2}{\sqrt{\pi }}\Gamma \left(\frac{1}{2}\right)={\sigma }^2\end{array}(1.4)$$

则方差 $var[x]=\mathbb{E}[x^2]-\mathbb{E}[x{]}^2={\sigma }^2$

补充知识：

1. 矩（moment）是对变量分布和形态特点的度量，$n$阶矩是指变量的n次方与其概率密度函数的乘积的积分，在文献中$n$阶矩通常用符号 ${\mu }_n$表示，直接使用变量计算的矩被称为原始矩（raw moment），移除均值后计算的矩被称为中心矩（central moment）。变量的一阶原始矩等价于数学期望，二至四阶中心矩被定义为方差，偏度和峰度。
   $$u_n^{\prime }=\int x^{n}P\left(x\right)dx\text{\hspace{1em}(1.5)}$$

2. $\Gamma$ 函数定义为 $\Gamma \left(x\right)={\int }_0^{\infty }{u}^{x-1}{e}^{-x}dx$，其具有两点重要性质：

   • $\Gamma (x+1)=x\Gamma (x)$
   • $\Gamma (\frac{1}{2})=\sqrt{\pi }$（可采用二重积分+夹逼定理求其平方值进行证明）

多元高斯分布的条件分布

假设 $\mathbf{x}$ 是一个服从高斯分布 $\mathcal{N}(\mathbf{x}|\mathbf{\mu },\mathbf{\Sigma })$的$D$维向量，把$\mathbf{x}$划分为不相交的子集${\mathbf{x}}_{\mathbf{a}},{\mathbf{x}}_{\mathbf{b}}$ ，有

$$\mathbf{x}=\left(\begin{array}{c}{\mathbf{x}}_a\\ {\mathbf{x}}_b\end{array}\right)$$

$$\mathbf{\mu }=\left(\begin{array}{c}{\mathbf{\mu }}_a\\ {\mathbf{\mu }}_b\end{array}\right)$$

$$\mathbf{\Sigma }=\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)$$

其中${\mathbf{\Sigma }}^T=\mathbf{\Sigma }$，则${\mathbf{\Sigma }}_{ba}^T={\mathbf{\Sigma }}_{ab}$，同时可以使用精度矩阵方便表达: $\mathbf{\Lambda }\equiv {\mathbf{\Sigma }}^{-1}$
为了求条件分布函数$p({\mathbf{x}}_a|{\mathbf{x}}_b)$，考虑概率的乘积问题，即 $p({\mathbf{x}}_a|{\mathbf{x}}_b)=\frac{p({\mathbf{x}}_a,{\mathbf{x}}_b)}{p({\mathbf{x}}_b)}$
为了方便计算，我们可以简化常数项的部分，即

$$\frac{p({\mathbf{x}}_a,{\mathbf{x}}_b)}{p({\mathbf{x}}_b)}=\frac{C_{{\mathbf{x}}_a,{\mathbf{x}}_b}}{C_{{\mathbf{x}}_b}}\exp \left(\frac{1}{2}{\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)}^{\mathbf{T}}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)-\frac{1}{2}\left({\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)}^{\mathbf{T}},{\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)}^{\mathbf{T}}\right){\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)}^{-1}\left(\begin{array}{c}{\mathbf{x}}_a-{\mathbf{\mu }}_a\\ {\mathbf{x}}_a-{\mathbf{\mu }}_a\end{array}\right)\right)(1.6)$$

重点分析函数内第二项，该项的核心部分为协方差矩阵的逆，由于${\mathbf{x}}_{\mathbf{b}}$是条件项，应尽可能保留${\mathbf{\Sigma }}_{bb}$，即可以通过初等变换简化该项形式，具体如下：

$$\left(\begin{array}{cc}\mathbf{I}& -{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\\ 0& \mathbf{I}\end{array}\right)\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)\left(\begin{array}{cc}\mathbf{I}& 0\\ -{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}& \mathbf{I}\end{array}\right)=\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}& 0\\ 0& {\mathbf{\Sigma }}_{bb}\end{array}\right)(1.7)$$

由形式可知，等号两边取逆：

$${\left(\begin{array}{cc}\mathbf{I}& 0\\ -{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}& \mathbf{I}\end{array}\right)}^{-1}{\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)}^{-1}{\left(\begin{array}{cc}\mathbf{I}& -{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\\ 0& \mathbf{I}\end{array}\right)}^{-1}=\left(\begin{array}{cc}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}& 0\\ 0& {\mathbf{\Sigma }}_{bb}^{-1}\end{array}\right)(1.8)$$

$${\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)}^{-1}=\left(\begin{array}{cc}\mathbf{I}& 0\\ -{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}& \mathbf{I}\end{array}\right)\left(\begin{array}{cc}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}& 0\\ 0& {\mathbf{\Sigma }}_{bb}^{-1}\end{array}\right)\left(\begin{array}{cc}\mathbf{I}& -{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\\ 0& \mathbf{I}\end{array}\right)(1.9)$$

此时：
$$\begin{gathered} -\frac{1}{2}\left({\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)}^{\mathbf{T}},{\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)}^{\mathbf{T}}\right){\left(\begin{array}{cc}{\mathbf{\Sigma }}_{aa}& {\mathbf{\Sigma }}_{ab}\\ {\mathbf{\Sigma }}_{ba}& {\mathbf{\Sigma }}_{bb}\end{array}\right)}^{-1}\left(\begin{array}{c}{\mathbf{x}}_a-{\mathbf{\mu }}_a\\ {\mathbf{x}}_b-{\mathbf{\mu }}_b\end{array}\right) \\ =-\frac{1}{2}\left({\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)}^{\mathbf{T}},{\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)}^{\mathbf{T}}\right)\left(\begin{array}{cc}\mathbf{I}& 0\\ -{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}& \mathbf{I}\end{array}\right)\left(\begin{array}{cc}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}& 0\\ 0& {\mathbf{\Sigma }}_{bb}^{-1}\end{array}\right)\left(\begin{array}{cc}\mathbf{I}& -{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\\ 0& \mathbf{I}\end{array}\right)\left(\begin{array}{c}{\mathbf{x}}_a-{\mathbf{\mu }}_a\\ {\mathbf{x}}_b-{\mathbf{\mu }}_b\end{array}\right) \\ =-\frac{1}{2}\left({\left({\mathbf{x}}_a-{\mathbf{\mu }}_a\right)}^{\mathbf{T}},{\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)}^{\mathbf{T}}\right)\left(\begin{array}{cc}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}& -{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\\ -{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}& {\mathbf{\Sigma }}_{bb}^{-1}+{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\end{array}\right)\left(\begin{array}{c}{\mathbf{x}}_a-{\mathbf{\mu }}_a\\ {\mathbf{x}}_b-{\mathbf{\mu }}_b\end{array}\right) \\ =-\frac{1}{2}{\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)}^{\mathbf{T}}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)-\frac{1}{2}{\left({\mathbf{x}}_a-{\mathbf{\mu }}_a-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)\right)}^{\mathbf{T}}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}\left({\mathbf{x}}_a-{\mathbf{\mu }}_a-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)\right)(1.10) \end{gathered}$$

因此：
$$\frac{p({\mathbf{x}}_a,{\mathbf{x}}_b)}{p({\mathbf{x}}_b)}=-\frac{1}{2}{\left({\mathbf{x}}_a-{\mathbf{\mu }}_a-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)\right)}^{\mathbf{T}}{\left({\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\right)}^{-1}\left({\mathbf{x}}_a-{\mathbf{\mu }}_a-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)\right)(1.11)$$

注意到条件概率分布其实是一个高斯分布，满足：
$${\mathbf{\mu }}_{a|b}={\mathbf{\mu }}_a-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}\left({\mathbf{x}}_b-{\mathbf{\mu }}_b\right)\text{\hspace{1em}(1.12)}$$

$${\mathbf{\Sigma }}_{a|b}={\mathbf{\Sigma }}_{aa}-{\mathbf{\Sigma }}_{ab}{\mathbf{\Sigma }}_{bb}^{-1}{\mathbf{\Sigma }}_{ba}\text{\hspace{1em}(1.13)}$$

其中可以看出均值是 ${\mathbf{x}}_b$ 的线性函数，协方差与其无关。该式说明已知协方差矩阵和 ${\mathbf{x}}_b$ 可以求出 ${\mathbf{x}}_a$ 的分布。