机器学习算法1_标准方程法

机器学习算法第一篇

主要内容:多项式回归之标准方程法算法展开

通俗的讲:

算法目的:

• 求通过训练集计算出一个最合适的线\面\超平面$h_{\theta }(x)$,使所有点到它的距离(误差)之和最小,
  将测试的特征向量带入该$h_{\theta }(x)$所得的值即为预测值

步骤:

1. 设一个目标函数(线\面\超平面) $h_{\theta }(x)={\theta }_1{x}_i^{(1)}+{\theta }_2{x}_i^{(2)}+...+{\theta }_n{x}_i^{(n)}$ ,一个n维特征带入后可得到预测值${\hat{y}}_i$

2. 设一个损失函数$J({\theta }_1,{\theta }_2...{\theta }_n)=\frac{1}{2m}{\sum }_{i=1}^m({\hat{y}}_i-y_i{)}^2$,它表示所有点预测值${\hat{y}}_i$到真实值y之间的误差的平方和的平均,数学上可以代表目标函数的好坏,即拟合度的高低

3. 所以我们目标可以理解为在学习阶段:求合适的$({\theta }_1,{\theta }_2...{\theta }_n)$使损失函数最小

4. 带入训练集数据,通过梯度下降法或标准方程法解出最合适的$({\theta }_1,{\theta }_2...{\theta }_n)$,然后带入线\面\超平面函数 $h_{\theta }(x)={\theta }_1{x}_i^{(1)}+{\theta }_2{x}_i^{(2)}+...+{\theta }_n{x}_i^{(n)}$ 得到目标函数(回归函数)

5. 把需要测试的数据输入该回归函数,可得到预测值(也可测试回归函数的拟合度,即参数的好坏)

展开来说:

1. 有训练集T={(x1,y1),(x2,y2),(x3,y3)...(xm,ym)}由P(X,Y)独立同分布产生T=\{(x_1,y_1),(x_2,y_2),(x_3,y_3)...(x_m,y_m) \}由P(X,Y)独立同分布产生T={(x1​,y1​),(x2​,y2​),(x3​,y3​)...(xm​,ym​)}由P(X,Y)独立同分布产生
   $$T:\left[\begin{array}{cccc}{x}_1^{(1)}& x_1^{(2)}...& x_1^{(n)}& y_1\\ x_2^{(1)}& x_2^{(2)}...& x_2^{(n)}& y_2\\ x_3^{(1)}& x_3^{(2)}...& x_3^{(n)}& y_3\\ .& ....& .& .\\ .& ....& .& .\\ x_m^{(1)}& x_m^{(2)}...& x_m^{(n)}& y_m\end{array}\right]$$

2. 输入空间$\mathcal{X}⊊R^{\ast }$为n维向量的集合

3. 输出空间为标记集合$\mathcal{Y}=R$,为全体实数标量集合

4. 输入为特征向量$x_i(x_i^{(1)},x_i^{(2)}...x_i^{(n)})\in \mathcal{X}$

5. 输出为类标记$y\in \mathcal{Y}$

6. X是定义在输入空间$\mathcal{X}$上的随机向量

7. Y是定义在输入空间$\mathcal{Y}$上的随机变量

8. 设回归函数 $h_{\theta }(x)={\theta }_1{x}_i^{(1)}+{\theta }_2{x}_i^{(2)}+...+{\theta }_n{x}_i^{(n)}$
   任意点$P_i(x_i^{(1)},x_i^{(2)}...x_i^{(n)})$带入该式可得到点$P_i$到预测值${\hat{y}}_i$

9. 带入多项式模型的损失函数$J({\theta }_1,{\theta }_2...{\theta }_n)=\frac{1}{2m}{\sum }_{i=1}^m({\hat{y}}_i-y_i{)}^2$

10. 即$$J({\theta }_1,{\theta }_2...{\theta }_n)=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x)-y_i{)}^2\text{\hspace{1em}(1)}$$

11. 在训练阶段, 代价函数的x 与y 都是已知量 $\theta$为变量量
    因此算法目标等价于求损失函数$J({\theta }_1,{\theta }_2...{\theta }_n)$的值取最小值时候的变量$({\theta }_1,{\theta }_2...{\theta }_n)$

12. 因为多项式模型的代价函数为凸函数,所以令其导数为0 可求出最合适的$({\theta }_1,{\theta }_2...{\theta }_n)$

求导过程:

由于对损失函数$J({\theta }_1,{\theta }_2...{\theta }_n)=\frac{1}{2m}{\sum }_{i=1}^m(h_{\theta }(x)-y_i{)}^2$的求导过程为较复杂.计算量较大庞大,
一般采用将式子转换矩阵,然后求解的方式

1. 子式矩阵化

• $\theta =\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ {\theta }_3\\ .\\ .\\ {\theta }_m\end{array}\right]$

• $J({\theta }_1,{\theta }_2...{\theta }_n)\Rightarrow J(\theta )$

• $y_i\Rightarrow Y=y\_data=\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\\ .\\ .\\ y_m\end{array}\right]$

• $x_i\Rightarrow X=x\_data=\left[\begin{array}{ccc}{x}_1^{(1)}& x_1^{(2)}...& x_1^{(n)}\\ x_2^{(1)}& x_2^{(2)}...& x_2^{(n)}\\ x_3^{(1)}& x_3^{(2)}...& x_3^{(n)}\\ .& ....& .\\ .& ....& .\\ x_m^{(1)}& x_m^{(2)}...& x_m^{(n)}\end{array}\right]$

• $h_{\theta }(x_i)\Rightarrow \left[\begin{array}{c}{h}_{\theta }(x_1)\\ h_{\theta }(x_2)\\ h_{\theta }(x_3)\\ .\\ .\\ h_{\theta }(x_m)\end{array}\right]=\left[\begin{array}{c}{\theta }_1{x}_1^{(1)}+{\theta }_2{x}_1^{(2)}+...+{\theta }_n{x}_1^{(n)}\\ {\theta }_1{x}_2^{(1)}+{\theta }_2{x}_2^{2)}+...+{\theta }_n{x}_2^{(n)}\\ {\theta }_1{x}_3^{(1)}+{\theta }_2{x}_3^{(2)}+...+{\theta }_n{x}_3^{(n)}\\ ....\\ ....\\ {\theta }_1{x}_m^{(1)}+{\theta }_2{x}_m^{(2)}+...+{\theta }_n{x}_m^{(n)}\end{array}\right]=X\cdot \theta$

• $\frac{\partial f}{\partial {\theta }_i}\Rightarrow \nabla J(\theta )=\left[\begin{array}{c}\frac{\partial f}{\partial {\theta }_1}\\ \frac{\partial f}{\partial {\theta }_2}\\ \frac{\partial f}{\partial {\theta }_3}\\ .\\ .\\ \frac{\partial f}{\partial {\theta }_n}\end{array}\right]$

正向转换

$$J({\theta }_1,{\theta }_2...{\theta }_n)=\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x_i)-y_i{)}^2$$

$$\Rightarrow J({\theta }_1,{\theta }_2...{\theta }_n)=\frac{1}{m}\left({\left(h_{\theta }(x_1)-y_1\right)}^2+{\left(h_{\theta }(x_2)-y_2\right)}^2+{\left(h_{\theta }(x_3)-y_3\right)}^2...+{\left(h_{\theta }(x_m)-y_m\right)}^2\right)$$
 \  
$$套公式{\left[\begin{array}{c}a\\ b\\ c\end{array}\right]}^T\left[\begin{array}{c}a\\ b\\ v\end{array}\right]=a^2+b^2+c^2$$

$$所以上上式可改为$$
$$\Rightarrow J(\theta )=\frac{1}{m}{\left[\begin{array}{c}{h}_{\theta }(x_1)-y_1\\ h_{\theta }(x_2)-y_2\\ h_{\theta }(x_3-y_3\\ ...\\ h_{\theta }(x_m)-y_m\end{array}\right]}^T\left[\begin{array}{c}{h}_{\theta }(x_1)-y_1\\ h_{\theta }(x_2)-y_2\\ h_{\theta }(x_3-y_3\\ ...\\ h_{\theta }(x_m)-y_m\end{array}\right]$$

$$\Rightarrow J(\theta )=\frac{1}{m}{\left(\left[\begin{array}{c}{h}_{\theta }(x_1)\\ h_{\theta }(x_2)\\ h_{\theta }(x_3\\ ...\\ h_{\theta }(x_m)\end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\\ ...\\ y_m\end{array}\right]\right)}^T\left(\left[\begin{array}{c}{h}_{\theta }(x_1)\\ h_{\theta }(x_2)\\ h_{\theta }(x_3)\\ ...\\ h_{\theta }(x_m)\end{array}\right]-\left[\begin{array}{c}{y}_1\\ y_2\\ y_3\\ ...\\ y_m\end{array}\right]\right)$$
 \  

$$将子式带入$$
$$J(\theta )=\frac{1}{m}(X\theta -Y{)}^T(X\theta -Y)$$
$$\Rightarrow J(\theta )=\frac{1}{m}[({\theta }^T{X}^{T}X\theta -{\theta }^T{X}^{T}Y-Y^{T}X\theta +Y^{T}Y]$$
  \  

$$\begin{gathered} 矩阵求导得 \\ \frac{dJ(\theta )}{d\theta }=\frac{1}{m}[2X^{T}X\theta -2X^{T}Y] \end{gathered}$$
  \  

$$\begin{gathered} 令上式等于0得 \\ \theta =(X^{T}X{)}^{-1}{X}^{T}Y \end{gathered}$$
  \  

$$将X=x\_data,Y=y\_data带入上式将得到目标\theta =\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ {\theta }_3\\ .\\ .\\ {\theta }_m\end{array}\right]$$