机器学习进阶（一）回归

线性回归

1. 目标函数
   $$J(\theta )=\frac{1}{2}\sum _{i=1}^n(h_{\theta }(x^{(i)})-y^{(i)}{)}^2=\frac{1}{2}(X\theta -y{)}^T(X\theta -y)$$
2. 求解方法
   正规求解：$$\theta =(X^{T}X{)}^{-1}{X}^{T}y$$
   梯度下降：$$\frac{\partial J(\theta )}{\partial {\theta }_j}={\theta }_j-\alpha \sum _{i=1}^n(h_{\theta }(x^{(i)})-y^{(i)})x_j$$
   3.正则项

\quad	L0正则项	L1正则项	L2正则项	Elastic Net
形式	∑ I { θ i ≠ 0 } \sum I_{\{\theta_i \neq0\}} ∑I{θi​​=0}​	$\sum |{\theta }_i|$	$\sum {\theta }_i^2$	$\rho \sum |{\theta }_i|+(1-\rho )\sum {\theta }_i^2,\rho \in [0,1]$
含义	L0正则化的值是模型参数中非零参数的个数	L1范数是指向量中各个元素绝对值之和	L2正则化标识各个参数的平方的和的开方值	Elastic Net则为L1和L2的加权组合
结果倾向		L1会趋向于产生少量的特征，而其他的特征都是0	L2会选择更多的特征，但这些特征都会接近于0
使用场景		在所有特征中只有少数特征起重要作用的情况下，选择L1比较合适。L1不仅可以作为正则化手段，其在特征选择时候非常有用	如果所有特征中，大部分特征都能起作用，而且起的作用很平均，那么使用L2更合适。

4. $R^2$ 评估
   $$R^2=\frac{{\sum }_{i=1}^n(\widehat{y_i}-\overline{y}{)}^2}{{\sum }_{i=1}^n(y_i-\overline{y}{)}^2}=1-\frac{{\widehat{ϵ}}_i^2}{{\sum }_{i=1}^n(y_i-\overline{y}{)}^2}=1-\frac{(\hat{y}-y_i{)}^2}{{\sum }_{i=1}^n(y_i-\overline{y}{)}^2}$$
   $$(\sum _{i=1}^n(y_i-\overline{y}{)}^2\ge \sum _{i=1}^n(\widehat{y_i}-\overline{y}{)}^2+\sum _{i=1}^n{\widehat{ϵ}}_i^2)$$
   其中，$\hat{y}$为估计值，$x_i,y_i$为样本值。$R^2$越大，拟合效果越好。括号中的等号当且仅当$\theta$为无偏估计时成立。

梯度下降

\quad	批量梯度下降	随机梯度下降	mini-batch 梯度下降
式子	${\theta }_j:{\theta }_j-\alpha {\sum }_{i=1}^n(h_{\theta }(x^{(i)})-y^{(i)})x_j$	${\theta }_j:{\theta }_j-\alpha (h_{\theta }(x^{(i)})-y^{(i)})x_j$	${\theta }_j:{\theta }_j-\alpha {\sum }_{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)})x_j$, $m<n$

局部加权线性回归

1. 目标函数
   $$J(\theta )=\sum _{i=1}^n{w}^{(i)}(h_{\theta }(x^{(i)})-y^{(i)}{)}^2$$
   $w$为权重，若为高斯函数，则$w^{(i)}=exp(-\frac{(x^{(i)}-x{)}^2}{2{\tau }^2})$，其中$\tau$为带宽。

logistic回归

1. 对数线性模型
   对数几率：$$\log it(p)=\log \frac{p}{1-p}=\log \frac{h_{\theta }(x)}{1-h_{\theta }(x)}={\theta }^{T}x$$

2. sigmoid函数
   $$g(z)=\frac{1}{1+e^{-z}}$$
   $$g^{\prime }(z)=g(z)(1-g(z))$$

3. 参数估计
   假定：$P(y=1|x;\theta )=h_{\theta }(x),P(y=0|x;\theta )=1-h_{\theta }(x)$，
   则有$p(y|x;\theta )=(h_{\theta }(x){)}^y(1-h_{\theta }(x){)}^{1-y}$
   那么对数似然函数为$$\log (L(\theta ))=\log \prod _{i=1}^{n}p(y|x;\theta )=\sum _{i=1}^n{y}^{(i)}\log h(x^{(i)})+(1-y^{(i)})\log (1-h(x^{(i)}))$$
   对${\theta }_j$求偏导
   $$\begin{array}{rl}\frac{\partial l(\theta )}{\partial {\theta }_j}& =\sum _{i=1}^n(\frac{y^{(i)}}{h(x^{(i)})}-\frac{1-y^{(i)}}{1-h(x^{(i)})})\frac{\partial h(x^{(i)})}{\partial {\theta }_j}\\ & =\sum _{i=1}^n(\frac{y^{(i)}}{g({\theta }^T{x}^{(i)})}-\frac{1-y^{(i)}}{1-g({\theta }^T{x}^{(i)})})\frac{\partial g({\theta }^T{x}^{(i)})}{\partial {\theta }_j}\\ & =\sum _{i=1}^n(\frac{y^{(i)}}{g({\theta }^T{x}^{(i)})}-\frac{1-y^{(i)}}{1-g({\theta }^T{x}^{(i)})})g({\theta }^T{x}^{(i)})(1-g({\theta }^T{x}^{(i)}))\frac{\partial {\theta }^T{x}^{(i)}}{\partial {\theta }_j}\\ & =\sum _{i=1}^n(y^{(i)}-g({\theta }^T{x}^{(i)}))x^{(i)}\end{array}$$

4. 损失函数
   $$L(\theta )=-\log (L(\theta ))$$

5. 梯度下降
   批量梯度下降
   $${\theta }_j:{\theta }_j+\alpha \sum _{i=1}^n(y^{(i)}-g({\theta }^T{x}^{(i)}))x^{(i)}$$
   随机梯度下降
   $${\theta }_j:{\theta }_j+\alpha (y^{(i)}-g({\theta }^T{x}^{(i)}))x^{(i)}$$
   mini-batch梯度下降
   $${\theta }_j:{\theta }_j+\alpha \sum _{i=1}^m(y^{(i)}-h_{\theta }(x^{(i)}))x^{(i)},m<n$$

\quad	线性回归	逻辑回归
$h_{\theta }(x^{(i)})$	$h_{\theta }(x^{(i)}))={\theta }^{T}x$	$h_{\theta }(x^{(i)}))=\frac{1}{1+e^{-x}}$
假设	$ϵ$(也就是$y={\theta }^{T}x+ϵ$)服从高斯分布，是指数族分布	$y$服从二项分布，是指数族分布

指数族分布的函数梯度下降都有类似的形式

Softmax回归

1. $softmax$函数
   $$softmax(z_k)=\frac{exp(z_k)}{{\sum }_{i=1}^{K}exp(z_i)}$$
   $$\begin{array}{rl}\frac{\partial softmax(z_k)}{\partial z_k}& =\frac{exp(z_k){\sum }_{i=1}^{K}exp(z_i)-exp(z_k)exp(z_k)}{({\sum }_{i=1}^{K}exp(z_i){)}^2}\\ & =softmax(z_k)(1-softmax(z_k))\end{array}$$
   $$\frac{\partial softmax(z_k)}{\partial z_j}=\frac{-exp(z_k)exp(z_j)}{({\sum }_{i=1}^{K}exp(z_i){)}^2}=-softmax(z_k)softmax(z_j)$$
   $$\frac{\partial \log softmax(z_k)}{\partial z_k}=\frac{\partial (z_k-\log {\sum }_{i=1}^{K}exp(z_i))}{\partial z_k}=1-softmax(z_k)$$
   $$\left(\frac{\partial \log softmax(z_k)}{\partial z_k}=\frac{1}{softmax(z_k)}\partial softmax(z_k)=1-softmax(z_k)\right)$$
2. $k$分类
   对于第$k$类，参数为${\theta }_k=({\theta }_1,\dots ,{\theta }_m{)}^T$，$m$为数据$x$的维数，$\Theta$是一个$k\times m$的矩阵。第$i$个样本$x^{(i)}$的标签${\mathbf{y}}^{(i)}=(y_1^{(i)},\dots ,y_k^{(i)})$
   则假定：
   $$\begin{array}{rl}P(y=k|x;\theta )& =\frac{exp({\theta }_k^{T}x)}{{\sum }_{k=1}^{K}exp({\theta }_k^{T}x)},k=1,2,\dots ,K\end{array}$$
   记$({\mathbf{y}}^{(i)}{)}^T=({\hat{y}}_1^{(i)},\dots ,{\hat{y}}_K^{(i)})=(P(y=1|x^{(i)};\theta ),\dots ,P(y=K|x^{(i)})$.
3. 对数似然函数$$\begin{array}{rl}\log (L(\theta ))& =\log \prod _{i=1}^{n}p(y|x;\theta )\\ & =log\prod _{i=1}^n\prod _{k=1}^K(P(y=k|x;\theta ){)}^{y_k^{(i)}}\\ & =log\prod _{i=1}^n\prod _{k=1}^K{\left(\frac{exp({\theta }_k^T{x}^{(i)})}{{\sum }_{k=1}^{K}exp({\theta }_k^T{x}^{(i)}))}\right)}^{y_k^{(i)}}\\ & \left(=\sum _{i=1}^n\sum _{i=1}^K{y}_k^{(i)}\log {\hat{y}}_k^{(i)}=\sum _{i=1}^n({\mathbf{y}}^{(i)}{)}^T\log \hat{\mathbf{y}}={\mathbf{y}}^T\log \hat{\mathbf{y}}\right)\\ & =\sum _{i=1}^n\sum _{k=1}^K{y}_k^{(i)}({\theta }_k^T{x}^{(i)}-\log \sum _{k=1}^{K}exp({\theta }_k^T{x}^{(i)})))\end{array}$$

对${\theta }_j$求偏导
$$\begin{array}{r}\frac{\partial L(\theta )}{\partial {\theta }_j}=\sum _{i=1}^n{y}_j^{(i)}(1-\frac{exp({\theta }_j^T{x}^{(i)})}{{\sum }_{l=1}^{K}exp({\theta }_l^T{x}^{(i)})})(x^{(i)}{)}^T\end{array}$$

3. 损失函数
   $$L(\theta )=-\log (L(\theta ))$$

4. 梯度下降
   批量梯度下降
   $${\theta }_j:{\theta }_j+\alpha \sum _{i=1}^n{y}_j^{(i)}(1-\frac{exp({\theta }_j^T{x}^{(i)})}{{\sum }_{l=1}^{K}exp({\theta }_l^T{x}^{(i)})})(x^{(i)}{)}^T$$
   随机梯度下降
   $${\theta }_j:{\theta }_j+\alpha y_j^{(i)}(1-\frac{exp({\theta }_j^T{x}^{(i)})}{{\sum }_{l=1}^{K}exp({\theta }_l^T{x}^{(i)})})(x^{(i)}{)}^T$$
   mini-batch梯度下降
   $${\theta }_j:{\theta }_j+\alpha \sum _{i=1}^m{y}_j^{(i)}(1-\frac{exp({\theta }_j^T{x}^{(i)})}{{\sum }_{l=1}^{K}exp({\theta }_l^T{x}^{(i)})})(x^{(i)}{)}^T,m<n$$