梯度下降法推导：线性回归

数据集格式

在机器学习里数据集格式一般如下:
第$i$个样本特征和标签写作：
$$\begin{gathered} x^i=(x_1^i,x_2^i,x_3^i,...,x_d^i{)}^T\in R^d \\ {y}^i\in R \end{gathered}$$
完整的数据集可以写作：
$$\begin{gathered} X=[x^1,x^2,\dots ,x^n] \\ =\left[\begin{array}{cccc}{x}_1^1& x_1^2& \dots & x_1^n\\ x_2^1& x_2^2& \dots & x_2^n\\ ⋮& ⋮& \ddots & ⋮\\ x_d^1& x_d^2& \dots & x_d^n\end{array}\right]\in R^{d\ast n} \\ y=[y^1,y^2,\dots ,y^n]\in R^n \end{gathered}$$

线性回归表达式

线性回归的权重$w$，默认写作
$$w=[w_1,w_2,\dots ,w_d{]}^T\in R^d$$
对于第$i$个样本，线性回归模型可写作：
$$\begin{gathered} f(x)=w^{T}x+b \\ =w_1{x}_1+w_2{x}_2+\dots +w_d{x}_d+b \end{gathered}$$
对于特征集合$X$，预测值$\hat{Y}$ 可以通过矩阵-向量乘法表示为
$$\hat{Y}=w^{T}X+b$$
如果以均方误差MSE （Mean Squared Error）作为损失函数的话，则损失函数(loss function or cost function)可以写作
$$\begin{gathered} L=\frac{1}{n}\sum _{i=1}^n(f(x^i)-y^i{)}^2 \\ =\frac{1}{n}\sum _{i=1}^n(w_1{x}_1^i+w_2{x}_2^i+\dots +w_d{x}_d^i+b-y^i{)}^2 \end{gathered}$$

通用范式

引入梯度符号$\nabla$，则
$$\begin{gathered} \nabla L(w)=\left[\begin{array}{c}\frac{\partial L}{\partial w_1}\\ \frac{\partial L}{\partial w_2}\\ ⋮\\ \frac{\partial L}{\partial w_d}\end{array}\right] \\ \nabla L(b)=[\frac{\partial L}{\partial b}]=\frac{\partial L}{\partial b} \end{gathered}$$
因此梯度下降一般化更新公式可写作
$$\begin{gathered} w\leftarrow w-\eta \nabla L(w) \\ b\leftarrow b-\eta \nabla L(b)=b-\eta \frac{\partial L}{\partial b} \end{gathered}$$
其中 $\eta$为学习率。

权重$w$的梯度下降

对于权重$w_j$，求梯度可得
$$\begin{gathered} \frac{\partial L}{\partial w_j}=\frac{2}{n}\sum _{i=1}^n(w_1{x}_1^i+w_2{x}_2^i+\dots +w_d{x}_d^i+b-y^i)x_j^i \\ =\frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)x_j^i \end{gathered}$$
则$w_j$的梯度下降更新可写作
$$\begin{gathered} w_j=w_j-\frac{\partial L}{\partial w_j} \\ =w_j-\frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)x_j^i \end{gathered}$$
同理，$w_1$,$w_2$,…$w_d$分别可写作
$$\begin{gathered} w_1=w_1-\frac{\partial L}{\partial w_1} \\ =w_1-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)x_1^i \\ {w}_2=w_2-\frac{\partial L}{\partial w_2} \\ =w_2-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)x_2^i \\ ⋮ \\ {w}_d=w_d-\frac{\partial L}{\partial w_2} \\ =w_d-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)x_d^i \end{gathered}$$
观察上述式子可知，$(f(x^i)-y^i)$为实数，而
$$\begin{gathered} w=[w_1,w_2,\dots ,w_d{]}^T\in R^d \\ {x}^i=(x_1^i,x_2^i,x_3^i,...,x_d^i{)}^T \end{gathered}$$
则权重$w$的梯度下降公式可写作
$$\begin{gathered} \left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_d\end{array}\right]=\left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_d\end{array}\right]-\eta \left[\begin{array}{c}\frac{2}{n}{\sum }_{i=1}^n(f(x^i)-y^i)x_1^i\\ \frac{2}{n}{\sum }_{i=1}^n(f(x^i)-y^i)x_2^i\\ ⋮\\ \frac{2}{n}{\sum }_{i=1}^n(f(x^i)-y^i)x_d^i\end{array}\right] \\ =\left[\begin{array}{c}{w}_1\\ w_2\\ ⋮\\ w_d\end{array}\right]-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)\left[\begin{array}{c}{x}_1^i\\ x_2^i\\ ⋮\\ x_d^i\end{array}\right] \end{gathered}$$
因此上述式子可写作
$$w=w-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)x^i$$
亦即
$$w=w-\eta \frac{2}{n}\sum _{i=1}^n(w^{T}x+b-y^i)x^i$$

权重$b$的梯度下降

同理可得权重$b$的梯度
$$\begin{gathered} \frac{\partial L}{\partial b}=\frac{2}{n}\sum _{i=1}^n(w_1{x}_1^i+w_2{x}_2^i+\dots +w_d{x}_d^i+b-y^i) \\ =\frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i) \end{gathered}$$
则梯度更新公式为
$$b=b-\eta \frac{\partial L}{\partial b}=b-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)$$
亦即：
$$b=b-\eta \frac{2}{n}\sum _{i=1}^n(f(x^i)-y^i)=b-\eta (w^{T}x+b-y^i)$$

参考文献

Zhang, Aston and Lipton, Zachary C. and Li, Mu and Smola, Alexander J , DiveintoDeepLearning