机器学习05：正则化

机器学习05：正则化

1 线性回归的正则化

1.1 损失函数

$$J(\theta )=\frac{1}{2m}\left[\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2+\lambda \sum _{j=0}^n{\theta }_j^2\right]$$

1.2 梯度下降法

$Repeat:$
$$\begin{gathered} {\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)} \\ \begin{array}{rl}{\theta }_j& :={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}-\frac{\lambda }{m}{\theta }_j\right]\\ & :={\theta }_j(1-\alpha \frac{\lambda }{m})-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}\end{array} \end{gathered}$$

1.3 正规方程法

$$X=\left[\begin{array}{c}(x^{(1)}{)}^T\\ \cdot \cdot \cdot \\ (x^{(m)}{)}^T\end{array}\right],y=\left[\begin{array}{c}{y}^{(1)}\\ \cdot \cdot \cdot \\ y^{(m)}\end{array}\right]$$
解得：
$$\theta =(X^{T}X+\lambda \left[\begin{array}{cccc}0& 0& \cdots & 0\\ 0& 1& \cdots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \cdots & 1\end{array}\right]{)}^{-1}{X}^{T}y$$

1.4 矩阵不可逆问题的解决

正则化之后，线性回归中的不可逆问题将不存在。

2 Logistic 回归的正则化

2.1 损失函数

$$J(\theta )=-\frac{1}{m}\left[\sum _{i=1}^m{y}^{(i)}log{h}_{\theta }(x^{(i)})+(1-y^{(i)})log(1-h_{\theta }(x^{(i)}))\right]+\frac{\lambda }{2m}\sum _{j=1}^n{\theta }_j^2$$

2.2 梯度下降

$Repeat:$
$$\begin{gathered} {\theta }_0:={\theta }_0-\alpha \frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_0^{(i)} \\ {\theta }_j:={\theta }_j-\alpha \left[\frac{1}{m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j^{(i)}-\frac{\lambda }{m}{\theta }_j\right] \end{gathered}$$
其中：
$$h_{\theta }(x)=\frac{1}{1-e^{-{\theta }^{T}x}}$$