机器学习 之 作业2

作业 1: 线性回归模型的极大似然估计

1. E{wMLE}E\{\textbf w_{MLE}\}E{wMLE​}

E{w^}=∫w^p(y∣X,w,δ2)dtE\{\widehat{\textbf w}\}=\int \widehat{\textbf w}p(y|\textbf X,w,\delta^2)dtE{w}=∫wp(y∣X,w,δ2)dt

其中$\hat{\text{w}}=({\text{X}}^T\text{X}{)}^{-1}{\text{X}}^T\text{y}$，带入
E{w^}=(XTX)−1XT∫yp(y∣X,w,δ2)dt=(XTX)−1XTE{y}=(XTX)−1XTXw^=w^E\{\widehat{\textbf w}\}=(\textbf X^T\textbf X)^{-1}\textbf X^T\int \textbf yp(y|\textbf X,w,\delta^2)dt\\ =(\textbf X^T\textbf X)^{-1}\textbf X^TE\{\textbf y\}\\ =(\textbf X^T\textbf X)^{-1}\textbf X^T\textbf X\widehat{\textbf w}\\ =\widehat{\textbf w}E{w}=(XTX)−1XT∫yp(y∣X,w,δ2)dt=(XTX)−1XTE{y}=(XTX)−1XTXw=w

2. Cov{wMLE}Cov\{\textbf w_{MLE}\}Cov{wMLE​}

cov{w^}=E{w^w^T}−E{w^}E{w^}T=E{w^w^T}−wwTcov\{\widehat{\textbf w}\}=E\{\widehat{\textbf w}\widehat{\textbf w}^T\}-E\{\widehat{\textbf w}\}E\{\widehat{\textbf w}\}^T\\ =E\{\widehat{\textbf w}\widehat{\textbf w}^T\}-\textbf w\textbf w^Tcov{w}=E{wwT}−E{w}E{w}T=E{wwT}−wwT

其中
E{w^w^T}=E{((XTX)−1XTy)((XTX)−1XTy)T}=(XTX)−1XTE{yyT}X(XTX)−1E\{\widehat{\textbf w}\widehat{\textbf w}^T\}=E\{((\textbf X^T\textbf X)^{-1}\textbf X^T\textbf y)((\textbf X^T\textbf X)^{-1}\textbf X^T\textbf y)^T\}\\ =(\textbf X^T\textbf X)^{-1}\textbf X^TE\{\textbf y\textbf y^T\}\textbf X(\textbf X^T\textbf X)^{-1}E{wwT}=E{((XTX)−1XTy)((XTX)−1XTy)T}=(XTX)−1XTE{yyT}X(XTX)−1

其中
cov{y}=δ2I=E{yyT}−E{y}E{y}TE{yyT}=E{y}E{y}T+δ2I=Xw(Xw)T+δ2I=XwwTXT+δ2Icov\{\textbf y\}=\delta^2I=E\{\textbf y\textbf y^T\}-E\{\textbf y\}E\{\textbf y\}^T\\ E\{\textbf y\textbf y^T\}=E\{\textbf y\}E\{\textbf y\}^T+\delta^2I\\ =\textbf X\textbf w(\textbf X\textbf w)^T+\delta^2I\\ =\textbf X\textbf w\textbf w^T\textbf X^T+\delta^2Icov{y}=δ2I=E{yyT}−E{y}E{y}TE{yyT}=E{y}E{y}T+δ2I=Xw(Xw)T+δ2I=XwwTXT+δ2I

因此
E{w^w^T}=(XTX)−1XTXwwTXTX(XTX)−1+δ2(XTX)−1XTX(XTX)−1=wwT+δ2(XTX)−1E\{\widehat{\textbf w}\widehat{\textbf w}^T\}=(\textbf X^T\textbf X)^{-1}\textbf X^T\textbf X\textbf w\textbf w^T\textbf X^T\textbf X(\textbf X^T\textbf X)^{-1}\\ +\delta^2(\textbf X^T\textbf X)^{-1}\textbf X^T\textbf X(\textbf X^T\textbf X)^{-1}\\ =\textbf w\textbf w^T+\delta^2(\textbf X^T\textbf X)^{-1}E{wwT}=(XTX)−1XTXwwTXTX(XTX)−1+δ2(XTX)−1XTX(XTX)−1=wwT+δ2(XTX)−1

cov{w^}=wwT+δ2(XTX)−1−wwT=δ2(XTX)−1cov\{\widehat{\textbf w}\}=\textbf w\textbf w^T+\delta^2(\textbf X^T\textbf X)^{-1}-\textbf w\textbf w^T\\ =\delta^2(\textbf X^T\textbf X)^{-1}cov{w}=wwT+δ2(XTX)−1−wwT=δ2(XTX)−1

作业 2: 关于岭回归与正则化

1. $求P(w|D)$的均值和方差

$$\begin{gathered} p(w|D)\simeq p(D|w)p(w) \\ =(\frac{1}{\sqrt{2\pi }\delta }{)}^n{e}^{-\frac{(\text{y}-\text{X}\text{w}{)}^T(\text{y}-\text{X}\text{w})}{2{\delta }^2\text{I}}}\times (\frac{1}{\sqrt{2\pi {\Sigma }_0}}{)}^n{e}^{-\frac{(\text{w}-{\mu }_0{)}^T(\text{w}-{\mu }_0)}{2{\Sigma }_0}} \\ \simeq e^{-\frac{1}{2}(\frac{(\text{y}-\text{X}\text{w}{)}^T(\text{y}-\text{X}\text{w})}{{\delta }^2\text{I}}+\frac{(\text{w}-{\mu }_0{)}^T(\text{w}-{\mu }_0)}{{\Sigma }_0})} \\ \simeq e^{-\frac{1}{2}(\frac{-2{\text{y}}^T\text{X}\text{w}+{\text{w}}^T{\text{X}}^T\text{X}\text{w}}{{\delta }^2\text{I}}+\frac{{\text{w}}^T\text{w}-2{\mu }_0^T\text{w}}{{\Sigma }_0})} \end{gathered}$$

后验概率应该为
$$\begin{gathered} p(\text{w}|D)=N({\mu }_w,{\Sigma }_w) \\ \simeq e^{-\frac{(\text{w}-{\mu }_w{)}^T(\text{w}-{\mu }_w)}{2{\Sigma }_w}} \\ \simeq e^{-\frac{{\text{w}}^T\text{w}-2{\mu }_w^T\text{w}}{2{\Sigma }_w}} \end{gathered}$$

这里面有一个线性二次项，对应相等，方差可解
$$\begin{gathered} \frac{{\text{w}}^T\text{w}}{{\Sigma }_w}=\frac{{\text{w}}^T{\text{X}}^T\text{X}\text{w}}{{\delta }^2\text{I}}+\frac{{\text{w}}^T\text{w}}{{\Sigma }_0} \\ ={\text{w}}^T(\frac{{\text{X}}^T\text{X}}{{\delta }^2\text{I}}+\frac{1}{{\Sigma }_0})\text{w} \\ \frac{1}{{\Sigma }_w}=\frac{{\text{X}}^T\text{X}}{{\delta }^2\text{I}}+\frac{1}{{\Sigma }_0} \\ {\Sigma }_w=(\frac{1}{{\delta }^2}{\text{X}}^T\text{X}+{\Sigma }_0^{-1}{)}^{-1} \end{gathered}$$

同样的，将线性一次项相等，会获得均值${\mu }_w$，这样$w$的方差，均值都获得了
$$\begin{gathered} \frac{-2{\mu }_w^T\text{w}}{{\Sigma }_w}=\frac{-2{\text{y}}^T\text{X}\text{w}}{{\delta }^2}+\frac{-2{\mu }_0^T\text{w}}{{\Sigma }_0} \\ \frac{{\mu }_w^T}{{\Sigma }_w}=\frac{{\text{y}}^T\text{X}}{{\delta }^2}+\frac{{\mu }_0^T}{{\Sigma }_0} \\ {\mu }_w^T=(\frac{{\text{y}}^T\text{X}}{{\delta }^2}+\frac{{\mu }_0^T}{{\Sigma }_0}){\Sigma }_w \\ {\mu }_w={\Sigma }_w(\frac{1}{{\delta }^2}{\text{X}}^T\text{y}+{\Sigma }_0^{-1}{\mu }_0) \\ =(\frac{1}{{\delta }^2}{\text{X}}^T\text{X}+{\Sigma }_0^{-1}{)}^{-1}(\frac{1}{{\delta }^2}{\text{X}}^T\text{y}+{\Sigma }_0^{-1}{\mu }_0) \end{gathered}$$

2. 为什么$w_{MAP}=(X^T+\lambda I{)}^{-1}{X}^{T}y?$

因为后验概率是高斯分布，$w$最有可能的地方就是在均值处，如果${\mu }_0=[0,0,...,0{]}^T$，那么就变成了$$w_{MAP}={\mu }_w=(\frac{1}{{\delta }^2}{\text{X}}^T\text{X}+{\Sigma }_0^{-1}{)}^{-1}\frac{1}{{\delta }^2}{\text{X}}^T\text{y}$$

基本相同。

3. 为什么$X^{T}X+\lambda I$是可逆的？

$({\text{X}}^T\text{X}{)}^T={\text{X}}^T\text{X}$，因此${\text{X}}^T\text{X}$是对称阵，明显的特点是对称轴都大于等于0，加上$\lambda I$后对称轴上都大于0，说明它为正定矩阵，正定矩阵可逆。