机器学习学习笔记--线性模型之一元线性回归

线性模型

一元线性回归

• 基本形式

$$f(\mathbf{x})=w_1{x}_1+w_2{x}_2+\dots +w_d{x}_d+b$$

向量形式
$$f(\mathbf{x})={\mathbf{w}}^{\mathrm{T}}\mathbf{x}+b$$
目标：均方误差最小化
$$\begin{array}{rl}\left(w^{\ast },b^{\ast }\right)& =\underset{(w,b)}{\mathrm{argmin}}\sum _{i=1}^m{\left(f\left(x_i\right)-y_i\right)}^2\\ & =\underset{(w,b)}{\mathrm{argmin}}\sum _{i=1}^m{\left(y_i-w{x}_i-b\right)}^2\end{array}$$
方法：线性回归模型的最小二乘“参数估计”。将$E_{w,b}$分别对$w$和$b$求导得到：
$$\frac{\partial E_{(w,b)}}{\partial w}=2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)$$

$$=\sum _{i=1}^m\frac{\partial }{\partial w}{\left(y_i-w{x}_i-b\right)}^2$$

$$=\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot \left(-x_i\right)$$

$$\frac{\partial E_{(w,b)}}{\partial b}=2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)$$

这里$E_{w,b}$是关于w和b的凸函数，当它关于w和b的导数均为零时，得到w和b的最优解.

判断凹凸性：

设f(x,y)在区域D上具有二阶连续偏导数，记$A = f_{xx}’’(x,y),B = f_{xy}’’(x,y),C = f_{yy}’’(x,y) $则：

（1)D上恒有A>0，且AC-$B^2>=0$时，f(x,y)在区域D上是凸函数；

（2）D上恒有$A<0$且$AC-B^2\ge 0$时，f(x,y)在区域D上是凹函数

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial w^2}& =\frac{\partial }{\partial w}\left(\frac{\partial E_{(w,b)}}{\partial w}\right)\\ & =\frac{\partial }{\partial w}\left[2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\right]\\ & =\frac{\partial }{\partial w}\left[2w\sum _{i=1}^m{x}_i^2\right]\end{array}$$

$$=2\sum _{i=1}^m{x}_i^2$$

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial w\partial b}& =\frac{\partial }{\partial b}\left(\frac{\partial E_{(w,b)}}{\partial w}\right)\\ & =\frac{\partial }{\partial b}\left[2\left(w\sum _{i=1}^m{x}_i^2-\sum _{i=1}^m\left(y_i-b\right)x_i\right)\right]\\ & =\frac{\partial }{\partial b}\left[-2\sum _{i=1}^m{y}_i{x}_i+2\sum _{i=1}^{m}b{x}_i\right)\\ & =\frac{\partial }{\partial b}\left(2\sum _{i=1}^{m}b{x}_i\right)\end{array}$$

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial b}& =\frac{\partial }{\partial b}\left[\sum _{i=1}^m{\left(y_i-w{x}_i-b\right)}^2\right]\\ & =\sum _{i=1}^m\frac{\partial }{\partial b}{\left(y_i-w{x}_i-b\right)}^2\\ & =\sum _{i=1}^{m}2\cdot \left(y_i-w{x}_i-b\right)\cdot (-1)\\ & =2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)\end{array}$$

$$\begin{array}{rl}\frac{{\partial }^2{E}_{(w,b)}}{\partial b^2}& =\frac{\partial }{\partial b}\left(\frac{\partial E_{(w,b)}}{\partial b}\right)\\ & =\frac{\partial }{\partial b}\left[2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)\right]\\ & =\frac{\partial }{\partial b}(2mb)\\ & =2m\end{array}$$

即：
$$A=2\sum _{i=1}^m{x}_i^{2}B=2\sum _{i=1}^m{x}_{i}C=2m$$

$$\begin{array}{rl}AC-B^2& =2m\cdot 2\sum _{i=1}^m{x}_i^2-{\left(2\sum _{i=1}^m{x}_i\right)}^2\\ & =4m\sum _{i=1}^m\left(x_i^2-x_i\overline{x}\right)\end{array}$$

又：
$$\sum _{i=1}^m{x}_i\overline{x}=\overline{x}\sum _{i=1}^m{x}_i=\overline{x}\cdot m\cdot \frac{1}{n}\cdot \sum _{i=1}^m{x}_i=m{\overline{x}}^2=\sum _{i=1}^m{\overline{x}}^2$$
所以上式
$$=4m\sum _{i=1}^m\left(x_i^2-x_i\overline{x}-x_i\overline{x}+x_i\overline{x}\right)=4m\sum _{i=1}^m\left(x_i^2-x_i\overline{x}-x_i\overline{x}+{\overline{x}}^2\right)=4m\sum _{i=1}^m{\left(x_i-\overline{x}\right)}^2$$

$$\begin{array}{rl}\frac{\partial E_{(w,b)}}{\partial b}& =2\left(mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)\right)=0\\ & mb-\sum _{i=1}^m\left(y_i-w{x}_i\right)=0\\ b& =\frac{1}{m}\sum _{i=1}^m\left(y_i-w{x}_i\right)\\ & =\overline{y}-w\overline{x}\end{array}$$

将$b=\overline{y}-w\overline{x}$代入$w{\sum }_{i=1}^m{x}_i^2={\sum }_{i=1}^m{y}_i{x}_i-{\sum }_{i=1}^{m}b{x}_i$可得
$$w\sum _{i=1}^m{x}_i^2-w\overline{x}\sum _{i=1}^m{x}_i=\sum _{i=1}^m{y}_i{x}_i-\overline{y}\sum _{i=1}^m{x}_i$$

$$w=\frac{{\sum }_{i=1}^m{y}_i{x}_i-\overline{y}{\sum }_{i=1}^m{x}_i}{{\sum }_{i=1}^m{x}_i^2-\overline{x}{\sum }_{i=1}^m{x}_i}$$

又有
$$\overline{y}\sum _{i=1}^m{x}_i=\frac{1}{m}\sum _{i=1}^m{y}_i\sum _{i=1}^m{x}_i=\overline{x}\sum _{i=1}^m{y}_i$$

$$\overline{x}\sum _{i=1}^m{x}_i=\frac{1}{m}\sum _{i=1}^m{x}_i\sum _{i=1}^m{x}_i=\frac{1}{m}{\left(\sum _{i=1}^m{x}_i\right)}^2$$

则
$$w=\frac{{\sum }_{i=1}^m{y}_i{x}_i-\overline{x}{\sum }_{i=1}^m{y}_i}{{\sum }_{i=1}^m{x}_i^2-\frac{1}{m}{\left({\sum }_{i=1}^m{x}_i\right)}^2}$$

二元凹凸函数求最值：

设f（x，y）是在开区域D内具有连续偏导数的凸（或者凹）函数，$\left(x_0,y_0\right)\in D$且$f_x^{\prime }\left(x_0,y_0\right)=0,f_y^{\prime }\left(x_0,y_0\right)=0$则$f\left(x_0,y_0\right)$必为f(x,y)在D内的最小值（或最大值）。

求解b:
$$b=\frac{1}{m}\sum _{i=1}^m\left(y_i-w{x}_i\right)$$
求解$w$:
$$w=\frac{{\sum }_{i=1}^m{y}_i\left(x_i-\overline{x}\right)}{{\sum }_{i=1}^m{x}_i^2-\frac{1}{m}{\left({\sum }_{i=1}^m{x}_i\right)}^2}$$
$w$的向量化：

将
$$\frac{1}{m}{\left(\sum _{i=1}^m{x}_i\right)}^2=\overline{x}\sum _{i=1}^m{x}_i=\sum _{i=1}^m{x}_i\overline{x}$$
代入分母得
$$\begin{array}{rl}w& =\frac{{\sum }_{i=1}^m{y}_i\left(x_i-\overline{x}\right)}{{\sum }_{i=1}^m{x}_i^2-{\sum }_{i=1}^m{x}_i\overline{x}}\\ & =\frac{{\sum }_{i=1}^m\left(y_i{x}_i-y_i\overline{x}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\overline{x}\right)}\end{array}$$
同理对分子分母做类似转换：
$$w=\frac{{\sum }_{i=1}^m\left(y_i{x}_i-y_i\overline{x}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\overline{x}\right)}=\frac{{\sum }_{i=1}^m\left(y_i{x}_i-y_i\overline{x}-y_i\overline{x}+y_i\overline{x}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\overline{x}-x_i\overline{x}+x_i\overline{x}\right)}$$

$$=\frac{{\sum }_{i=1}^m\left({\hat{y}}_i{x}_i-y_i\overline{x}-x_i\overline{y}+\overline{x}\overline{y}\right)}{{\sum }_{i=1}^m\left(x_i^2-x_i\overline{x}-x_i\overline{x}+{\overline{x}}^2\right)}=\frac{{\sum }_{i=1}^m\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{{\sum }_{i=1}^m{\left(x_i-\overline{x}\right)}^2}$$

令
$$\begin{array}{c}\mathbf{x}={\left(x_1,x_2,\dots ,x_m\right)}^T\mathbf{y}={\left(y_1,y_2,\dots ,y_m\right)}^T\\ {\mathbf{x}}_d={\left(x_1-\overline{x},x_2-\overline{x},\dots ,x_m-\overline{x}\right)}^T{\mathbf{y}}_d={\left(y_1-\overline{y},y_2-\overline{y},\dots ,y_m-\overline{y}\right)}^T\end{array}$$

$$\begin{array}{rl}w& =\frac{{\sum }_{i=1}^m\left(x_i-\overline{x}\right)\left(y_i-\overline{y}\right)}{{\sum }_{i=1}^m{\left(x_i-\overline{x}\right)}^2}\\ & =\frac{{\mathbf{x}}_d^T{\mathbf{y}}_d}{{\mathbf{x}}_d^T{\mathbf{x}}_d}\end{array}$$