主题模型（2）

1.LDA推导
   (2)似然概率
    一个词${\mathrm{W}}_{\mathrm{m}\mathrm{n}}$初始化为一个词t的概率为
$$p\left(w_{m,n}=t|{\vec{\vartheta }}_m,\underline{\Phi }\right)=\sum _{k=1}^{K}p\left(w_{m,n}=t|{\vec{\phi }}_k\right)p\left(z_{m,n}=k|{\vec{\vartheta }}_m\right)$$
    每个文档中出现主题k的概率乘以主题k下出现词t的概率，然后枚举所有主题求和得到，整个文档集合的似然函数为：
$$p(\mathcal{W}|\underline{\Theta },\underline{\Phi })=\prod _{m=1}^{M}p\left({\vec{w}}_m|{\vec{\vartheta }}_m,\underline{\Phi }\right)=\prod _{m=1}^M\prod _{n=1}^{N_m}p\left(w_{m,n}|{\vec{\vartheta }}_m,\underline{\Phi }\right)$$
   (3)Gibbs 采样
     每次选取概率向量的一个维度，给定其他维度的变量值采样当前维度的值，不断迭代输出待估计的参数；
     初始时随机给文本中每个词分配主题$Z^{(0)}$，然后统计每个主题z下出现词t的数量以及每个文档m下出现主题z的数量，每一轮计算$\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}|{\mathrm{z}}_{-\mathrm{i}},\mathrm{d},\mathrm{w}\right)$，即排除当前词的主题分布：根据所有其他词的主题分布估计当前词分配各个主题的概率；
     当得到当前属于所有主题的z的概率分布以后，根据这个概率分布为该词采样一个新的主题；
     用同样的方法更新下一个词的主题，直到发现每个文档的主题分布${\theta }_i$和每个主题的词分布${\phi }_j$收敛，算法停止，输出待估计的参数$\theta$和$\varphi$,同时每个单词的主题分布$Z_{\mathrm{mn}}$也得到；
     实际过程中，会设置最大迭代次数。每一次计算$\mathrm{p}\left({\mathrm{z}}_{\mathrm{i}}|{\mathrm{z}}_{-\mathrm{i}},\mathrm{d},\mathrm{w}\right)$公式称为Gibbs Updating Rule。
   (4)联合分布
$$p(\vec{w},\vec{z}|\vec{\alpha },\vec{\beta })=p(\vec{w}|\vec{z},\vec{\beta })p(\vec{z}|\vec{\alpha })$$
$$\begin{array}{rl}p(\vec{w}|\vec{z},\vec{\beta })& =\int p(\vec{w}|\vec{z},\underline{\Phi })p(\underline{\Phi }|\vec{\beta })\mathrm{d}{\underline{\Phi }}^{(\hat{\beta })}\\ & =\int \prod _{z=1}^K\frac{1}{\Delta (\vec{\beta })}\prod _{t=1}^V{\phi }_{z,t}^{n_z^{(t)}+{\beta }_t-1}\mathrm{d}{\vec{\phi }}_z\\ & =\prod _{z=1}^K\frac{\Delta \left({\vec{n}}_z+\vec{\beta }\right)}{\Delta (\vec{\beta })},{\vec{n}}_z={\left\{n_z^{(t)}\right\}}_{t=1}^V\end{array}$$
$$\begin{array}{rl}p(\vec{z}|\vec{\alpha })& =\int p(\vec{z}|\underline{\Theta })p(\underline{\Theta }|\vec{\alpha })\mathrm{d}\underline{\Theta }\\ & =\int \prod _{m=1}^M\frac{1}{\Delta (\vec{\alpha })}\prod _{k=1}^K{\vartheta }_{m,k}^{n_m^{(k)}+{\alpha }_k-1}\mathrm{d}{\vec{\vartheta }}_m\\ & =\prod _{m=1}^M\frac{\Delta \left({\vec{n}}_m+\vec{\alpha }\right)}{\Delta (\vec{\alpha })},{\vec{n}}_m={\left\{n_m^{(k)}\right\}}_{k=1}^K\end{array}$$
   (4)Gibbs Updating Rule
$$\begin{array}{rl}p\left(z_i=k|{\vec{z}}_{\neg i},\vec{w}\right)=& \frac{p(\vec{w},\vec{z})}{p\left(\vec{w},{\vec{z}}_{\neg i}\right)}=\frac{p(\vec{w}|\vec{z})}{p\left({\vec{w}}_{\neg i}|{\vec{z}}_{\neg i}\right)p\left(w_i\right)}\cdot \frac{p(\vec{z})}{p\left({\vec{z}}_{\neg i}\right)}\\ & \propto \frac{\Delta \left({\vec{n}}_z+\vec{\beta }\right)}{\Delta \left({\vec{n}}_{z,i}+\vec{\beta }\right)}\cdot \frac{\Delta \left({\vec{n}}_m+\vec{\alpha }\right)}{\Delta \left({\vec{n}}_{m,-i}+\vec{\alpha }\right)}\\ & =\frac{\Gamma \left(n_k^{(t)}+{\beta }_t\right)\Gamma \left({\sum }_{t=1}^V{n}_{k,\neg i}^{(t)}+{\beta }_t\right)}{\Gamma \left(n_{k,-i}^{(t)}+{\beta }_t\right)\Gamma \left({\sum }_{t=1}^V{n}_k^{(t)}+{\beta }_t\right)}\cdot \frac{\Gamma \left(n_m^{(k)}+{\alpha }_k\right)\Gamma \left({\sum }_{k=1}^K{n}_{m,-i}^{(k)}+{\alpha }_k\right)}{\Gamma \left(n_{m,\neg i}^{(k)}+{\alpha }_k\right)\Gamma \left({\sum }_{k=1}^K{n}_m^{(k)}+{\alpha }_k\right)}\\ & =\frac{n_{k,-i}^{(t)}+{\beta }_t}{{\sum }_{t=1}^V{n}_{k,+i}^{(t)}+{\beta }_t}\cdot \frac{n_{m,-i}^{(k)}+{\alpha }_k}{\left[{\sum }_{k=1}^K{n}_m^{(k)}+{\alpha }_k\right]-1}\\ & \propto \frac{n_{k,\neg i}^{(t)}+{\beta }_t}{{\sum }_{t=1}^V{n}_{k,\neg i}^{(t)}+{\beta }_t}\left(n_{m,-i}^{(k)}+{\alpha }_k\right)\end{array}$$
   (5)词分布和主题分布
$$\begin{array}{rl}{\phi }_{k,t}& =\frac{n_k^{(t)}+{\beta }_t}{{\sum }_{t=1}^V{n}_k^{(t)}+{\beta }_t}\\ {\vartheta }_{m,k}& =\frac{n_m^{(k)}+{\alpha }_k}{{\sum }_{k=1}^K{n}_m^{(k)}+{\alpha }_k}\end{array}$$
p ( ϑ ⃗ m ∣ x ⃗ m , α ⃗ ) = 1 Z ϑ m ∏ n = 1 N m p ( z m , n ∣ ϑ ⃗ m ) ⋅ p ( ϑ ⃗ m ∣ α ⃗ ) = Dir ⁡ ( ϑ ⃗ m ∣ n ⃗ m + α ⃗ ) p ( φ ⃗ k ∣ z ⃗ , w ⃗ , β ⃗ ) = 1 Z φ k ∏ { i : z i = k } p ( w i ∣ φ ⃗ k ) ⋅ p ( φ ⃗ k ∣ β ⃗ ) = Dir ⁡ ( φ ⃗ k ∣ n ⃗ k + β ⃗ ) \begin{aligned} p\left(\vec{\vartheta}_{m} | \vec{x}_{m}, \vec{\alpha}\right) &=\frac{1}{Z_{\vartheta_{m}}} \prod_{n=1}^{N_{m}} p\left(z_{m, n} | \vec{\vartheta}_{m}\right) \cdot p\left(\vec{\vartheta}_{m} | \vec{\alpha}\right)=\operatorname{Dir}\left(\vec{\vartheta}_{m} | \vec{n}_{m}+\vec{\alpha}\right) \\ p\left(\vec{\varphi}_{k} | \vec{z}, \vec{w}, \vec{\beta}\right) &=\frac{1}{Z_{\varphi_{k}}} \prod_{\left\{i: z_{i}=k\right\}} p\left(w_{i} | \vec{\varphi}_{k}\right) \cdot p\left(\vec{\varphi}_{k} | \vec{\beta}\right)=\operatorname{Dir}\left(\vec{\varphi}_{k} | \vec{n}_{k}+\vec{\beta}\right) \end{aligned} p(ϑ m​∣x m​,α )p(φ ​k​∣z ,w ,β ​)​=Zϑm​​1​n=1∏Nm​​p(zm,n​∣ϑ m​)⋅p(ϑ m​∣α )=Dir(ϑ m​∣n m​+α )=Zφk​​1​{i:zi​=k}∏​p(wi​∣φ ​k​)⋅p(φ ​k​∣β ​)=Dir(φ ​k​∣n k​+β ​)​