标准方程的证明

标准方程的证明

线性回归模型公式(第i个实例的预测值$\widehat{y_i}$):
$$\widehat{y_i}={\theta }_0+{\theta }_1{x}_{i,1}+{\theta }_2{x}_{i,2}+...+{\theta }_n{x}_{i,n}$$
转化成矩阵:
$$\widehat{y_i}=\left[\begin{array}{ccccc}1& x_{i,1}& x_{i,2}& \cdots & x_{i,n}\end{array}\right]\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ {\theta }_2\\ ⋮\\ {\theta }_n\end{array}\right]$$
简化为:
$$\widehat{y_i}={{\mathbf{x}}_{\mathbf{i}}}^T\theta$$
误差公式为:
$$MSE(\theta )=\frac{1}{m}\sum _{i=1}^m({\hat{y}}_i-y_i{)}^2=\frac{1}{m}\sum _{i=1}^m({{\mathbf{x}}_{\mathbf{i}}}^T\theta -y_i{)}^2$$
设:
$$\mathbf{c}=\left[\begin{array}{c}{{\mathbf{x}}_1}^T\theta -y_1\\ {{\mathbf{x}}_2}^T\theta -y_2\\ ⋮\\ {{\mathbf{x}}_{\mathbf{m}}}^T\theta -y_m\end{array}\right]=\left[\begin{array}{c}{{\mathbf{x}}_1}^T\theta \\ {{\mathbf{x}}_2}^T\theta \\ ⋮\\ {{\mathbf{x}}_{\mathbf{m}}}^T\theta \end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_m\end{array}\right]=\left[\begin{array}{c}{{\mathbf{x}}_1}^T\\ {{\mathbf{x}}_2}^T\\ ⋮\\ {{\mathbf{x}}_{\mathbf{m}}}^T\end{array}\right]\theta -\mathbf{y}=\mathbf{X}\theta -\mathbf{y}$$

则:
$$MSE(\theta )=\frac{1}{m}{\Vert \mathbf{c}\Vert }^2=\frac{1}{m}{\Vert \mathbf{X}\theta -\mathbf{y}\Vert }^2$$

$MSE(\theta )$要取到最小值,则对$MSE(\theta )=MSE({\theta }_0,{\theta }_1,\cdots ,{\theta }_n)=E$,相当于求解该多变量函数梯度为0的点,梯度向量为E函数对$\theta$的偏导数:
$$\frac{\partial E}{\partial \theta }=\left[\begin{array}{ccccc}\frac{\partial E}{\partial {\theta }_0}& \frac{\partial E}{\partial {\theta }_1}& \cdots & \frac{\partial E}{\partial {\theta }_n}& \end{array}\right]$$
由矩阵的求导法则及下一节证明出的公式可证:

设$g(\theta )=\mathbf{X}\theta -\mathbf{y}=\mathbf{u}$,则
$f(\mathbf{u})=MSE(\theta )=\frac{1}{m}{\Vert g(\theta )\Vert }^2=\frac{1}{m}{\Vert \mathbf{u}\Vert }^2$
$$\frac{\partial MSE(\theta )}{\partial \theta }=\frac{\partial f(\mathbf{u})}{\partial \theta }=\frac{\partial f(\mathbf{u})}{\partial \mathbf{u}}\frac{\partial \mathbf{u}}{\partial \theta }=\frac{\partial \frac{1}{m}{\Vert \mathbf{u}\Vert }^2}{\partial \mathbf{u}}\frac{\partial \mathbf{X}\theta -\mathbf{y}}{\partial \theta }=\frac{1}{m}\frac{\partial {\mathbf{u}}^T\mathbf{u}}{\partial \mathbf{u}}\mathbf{X}=\frac{2}{m}{\mathbf{u}}^T\mathbf{X}$$

则求解梯度全为0时$\theta$的值$\hat{\theta }$:
$$\frac{2}{m}{\left(\mathbf{X}\hat{\theta }-\mathbf{y}\right)}^T\mathbf{X}=0$$
$${\hat{\theta }}^T{\mathbf{X}}^T\mathbf{X}-{\mathbf{y}}^T\mathbf{X}=0$$
$${\hat{\theta }}^T={\mathbf{y}}^T\mathbf{X}{\left({\mathbf{X}}^T\mathbf{X}\right)}^{-1}$$
$$\hat{\theta }={\left({\mathbf{X}}^T\mathbf{X}\right)}^{-1}{\mathbf{X}}^T\mathbf{y}$$

本质上来说是矩阵求导的应用，特殊多项式求最小值,该计算涉及到求逆操作，对n×n矩阵的求逆的计算复杂度通常为$O(n^{2.4})$到$O(n^3)$之间，当特征数量比较大时(例如100000时),标准方程的计算会极其缓慢