机器学习算法5_逻辑回归

机器学习算法第四篇

1. 本文目内容:逻辑回归算法推导
2. 本文基于梯度下降法, 多元线性回归
3. 本文为纯数学向分析

各种预设

 \  
$$设有数据Data如下(上标为列号,下标为行号),y取值为1或0,用于区分该样本的分类$$
$$data=\left[\begin{array}{cccc}{x}_1^1& x_2^1...& x_n^1& y^1\\ x_1^2& x_2^2...& x_n^2& y^2\\ x_1^3& x_2^3...& x_n^3& y^3\\ .& ....& .& .\\ .& ....& .& .\\ x_1^m& x_2^m...& x_n^m& y^m\end{array}\right]$$

$$令x\_data=X=\left[\begin{array}{ccc}{x}_1^1& x_2^1...& x_n^1\\ x_1^2& x_2^2...& x_n^2\\ x_1^3& x_2^3...& x_n^3\\ .& ....& .\\ .& ....& .\\ x_1^m& x_2^m...& x_n^m\end{array}\right]\text{\hspace{1em}(1)}$$

$$令y\_data=Y=\left[\begin{array}{c}{y}^1\\ y^2\\ y^3\\ .\\ .\\ y^m\end{array}\right],y\in (1,0)\text{\hspace{1em}(2)}$$

 \  
$$设决策边界\to L:{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3+...+{\theta }_n{x}_n=0\text{\hspace{1em}(3)}$$
 \  
$$设函数\to \beta (x):{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3+...+{\theta }_n{x}_n\text{\hspace{1em}(4)}$$
$$则\to \Theta =\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\\ {\theta }_3\\ .\\ .\\ {\theta }_m\end{array}\right]\text{\hspace{1em}(5)}$$
$$理论1:样本点P(x_1,x_2...x_n)带入(2)式得到\beta (x)的函数值d,则d与样本点P到决策边界的距离成正比\text{\hspace{1em}(6)}$$
 \  
$$\begin{gathered} 理论2:设样本点P_1,P_2,P_3,...P_m,分别带入函数\beta (x)得到d_1,d_2,d_3,...d_m, \\ 则各点可依据d值得正负分为两部分, \\ 我们令d>0的样本点的标签值为1,使d<0的样本点的标签值为0, \\ 从而预测并分类这两类样本\text{\hspace{1em}(7)} \end{gathered}$$
 \  
$$\begin{gathered} 设sigmoid函数sigmoid(x)=\frac{1}{1+e^{-x}} \\ 该函数有特性: \\ 当输入值x大于0时输出(0.5,1)中的某值,并且输入的x越大,输出的值越是接近1, \\ 当当输入值x小于0时输出(0,0.5)中的某值,并且输入的x越小,输出的值越是接近0\text{\hspace{1em}(8)} \end{gathered}$$
 \  

算法过程:

$$\begin{array}{rl}& 1.在训练阶段通过不断给(2)中的函数\beta (x):{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3+...+{\theta }_n{x}_n\\ & 带入特征数据X,从而得到值d_1,d_2,d_3,...d_m\\ & 2.我们令将d_1,d_2,d_3,...d_m带入预测函数sigmoid函数(8)\\ & 从而得到Y_{预测}=(y^1,y^2,y^3,...y^m),预测并分类这两类样本\\ & 3.构建合适的代价函数J(\theta ),表达预测与真值差异的大小\\ & 4.对代价函数用梯度下降法求得最合适的({\theta }_1,{\theta }_2,...{\theta }_n,)\\ & 5.在测试阶段,将特征数据X_{测试}带入由{\theta }_{最佳}({\theta }_1,{\theta }_2,...{\theta }_n,)所组成的模型(预测函数):h_{\theta }(x)\\ & 得到的最佳预测值Y_{最佳预测}=(y_1,y_2...y_m),\\ & 6.依据y_i的值检测预测准确率,该值越接近1或0,则预测越准确,越接近0.5则预测越无效\end{array}\text{\hspace{1em}(8)}$$

目标:构建的合适的代价函数

ps:线性回归的代价函数直接套用是不合适的,因其在此场景为非凸函数
 \  
$$\begin{gathered} 因上述算法过程我们知道当\sum _{i=1}^m|d_i|的值越大,求得的决策边界越合适,越泾渭分明,越居中 \\ 因此我们可以构建函数的目标:求合适的({\theta }_1,{\theta }_2,...{\theta }_n,)使函数\sum _{i=1}^m|d_i|的值趋向于0或1\text{\hspace{1em}(9)} \end{gathered}$$
 \  
$$\begin{gathered} 构建预测函数反应传入样本点的预测值 \\ {h}_{\theta }(x)=sigmoid[(\beta (x^i)]=\frac{1}{1-e^{\beta (x^i)}} \\ 根据(8)式所述,该h_{\theta }(x)函数将原\beta (x)函数的值域映射成(0,1)区间 \\ 并继承(8)式的特性 \\ 当输入值x大于0时输出(0.5,1)中的某值,并且输入的x越大,输出的值越是接近1, \\ 当当输入值x小于0时输出(0,0.5)中的某值,并且输入的x越小,输出的值越是接近0 \end{gathered}$$
 \  
$$故(9)式的目标可等价于:求合适的({\theta }_1,{\theta }_2,...{\theta }_n)使函数h_{\theta }(x^{(i)})的值趋向于0或1\text{\hspace{1em}(10)}$$
 \  
$$构建函数Cost(\theta ,y)=\{\begin{array}{ll}-log(h_{\theta }(x^{(i)}),& \text{if yi is 1}\\ -log(1-h_{\theta }(x^{(i)}),& \text{if yi is 0}\end{array}\text{\hspace{1em}(11)}$$

$$合并得Cost(\theta ,y)=-y^{i}log[h_{\theta }(x^{(i)}]-(1-y^i)log[1-h_{\theta }(x^{(i)})]\text{\hspace{1em}(12)}$$
 \  
$$\begin{gathered} (8)式的上式函数为在定义域(0,1)的单调减函数,具有随函数值趋向0,输入值趋向于1的特性, \\ (8)该式的下式函数为在定义域(0,1)的单调增函数,具有随函数值趋向0,输入值趋向于0的特性 \\ 因此(12)式具有特性:,当值域趋向于0的时候输入值趋向于0或1 \\ 即(10)式的目标可等价于:合适的({\theta }_1,{\theta }_2,...{\theta }_n)使函数J(\theta ,y)的值趋向于0 \\ 又同时因为该式为凸函数所以(12)式非常适合做为代价函数 \end{gathered}$$

 \  
$$\begin{gathered} 于是我们定义代价函数 \\ Cost(\theta ,y)=-\sum _{i=1}^m\left\{y^{i}log[h_{\theta }(x^{(i)})]+(1-y^i)log[1-h_{\theta }(x^{(i)})]\right\}\text{\hspace{1em}(12)} \end{gathered}$$

梯度下降

 \  
$$\begin{gathered} 由于代价函数（12）式为复合函数，复合了下面两条式子 \\ {h}_{\theta }(x)=sigmoid[(\beta (x^i)]=\frac{1}{1-e^{\beta (x^i)}} \\ \text{\hspace{1em}(13)} \end{gathered}$$
$$\beta (x):{\theta }_1{x}_1+{\theta }_2{x}_2+{\theta }_3{x}_3+...+{\theta }_n{x}_n\text{\hspace{1em}(14)}$$
 \  
$$\begin{gathered} 所以结合（13）（14）对复合函数（12）求导得： \\ \frac{\partial f}{\partial {\theta }_1}=-\sum _{i=1}^m\left\{\frac{y^i}{h_{\theta }(x^{(i)})}+\frac{(1-y^i)}{1-h_{\theta }(x^{(i)})}\right\}\frac{\partial h_{\theta }(x^{(i)})}{\partial {\theta }_1}\text{\hspace{1em}(15)} \end{gathered}$$
$$=-\sum _{i=1}^m\left\{\frac{y^i}{h_{\theta }(x^{(i)})}-\frac{(1-y^i)}{1-h_{\theta }(x^{(i)})}\right\}{h}_{\theta }^{‘}(x^{(i)})x_1^i$$

$$=-\sum _{i=1}^m\left\{\frac{y^i-h_{\theta }(x^{(i)})}{h_{\theta }(x^{(i)})[1-h_{\theta }(x^{(i)})]}\right\}{h}_{\theta }^{‘}(x^{(i)})x_1^i$$
$$又h_{\theta }^{‘}(x^{(i)})=h_{\theta }(x^{(i)})[1-h_{\theta }(x^{(i)})]求导过程略$$

$$所以\frac{\partial f}{\partial {\theta }_1}=-\sum _{i=1}^m\left\{x_1^i[y^i-h_{\theta }(x^{(i)})]\right\}\text{\hspace{1em}(16)}$$

 \  

$$\begin{gathered} 同理可得到 \\ \frac{\partial f}{\partial {\theta }_1}=-\sum _{i=1}^m\left\{x_1^i[y^i-h_{\theta }(x^{(i)})]\right\} \\ \frac{\partial f}{\partial {\theta }_2}=-\sum _{i=1}^m\left\{x_2^i[y^i-h_{\theta }(x^{(i)})]\right\} \\ \frac{\partial f}{\partial {\theta }_3}=-\sum _{i=1}^m\left\{x_3^i[y^i-h_{\theta }(x^{(i)})]\right\} \\ ... \\ \frac{\partial f}{\partial {\theta }_n}=-\sum _{i=1}^m\left\{x_n^i[y^i-h_{\theta }(x^{(i)})]\right\}\text{\hspace{1em}(17)} \end{gathered}$$

 \  

$$各\theta 用其偏导数乘以合适的学习率lr后更新自己$$

 \  

$$\begin{gathered} {\theta }_1={\theta }_1-lr\frac{\partial f}{\partial {\theta }_1} \\ {\theta }_2={\theta }_2-lr\frac{\partial f}{\partial {\theta }_2} \\ {\theta }_3={\theta }_3-lr\frac{\partial f}{\partial {\theta }_3} \\ ... \\ {\theta }_n={\theta }_n-lr\frac{\partial f}{\partial {\theta }_n}\text{\hspace{1em}(18)} \end{gathered}$$

 \  

$$重复(17)-(18)步足够的次数即可得到最优的({\theta }_1,{\theta }_2,...{\theta }_n)使得代价函数(12)取最小值$$

 \  

矩阵化

由于常规做法计算复杂,我们一般用矩阵的方式做梯度下降,以减少计算量与复杂度

• 首先把(17)式中用到的子式矩阵化

 \  
$$\beta \Rightarrow \left[\begin{array}{c}\beta (x^1)\\ \beta (x^2)\\ \beta (x^3)\\ .\\ .\\ \beta (x^m)\end{array}\right]=\left[\begin{array}{c}{\theta }_1{x}_1^1+{\theta }_2{x}_2^1+...+{\theta }_n{x}_n^1\\ {\theta }_1{x}_1^2+{\theta }_2{x}_2^2+...+{\theta }_n{x}_n^2\\ {\theta }_1{x}_1^3+{\theta }_2{x}_2^3+...+{\theta }_n{x}_n^3\\ ....\\ ....\\ {\theta }_1{x}_1^m+{\theta }_2{x}_2^m+...+{\theta }_n{x}_n^m\end{array}\right]=X\cdot \theta$$

 \  
$$\begin{gathered} H=\left[\begin{array}{c}{h}_{\theta }(x^1)\\ h_{\theta }(x^2)\\ h_{\theta }(x^3)\\ .\\ .\\ h_{\theta }(x^m)\end{array}\right]=sigmoid(\beta ) \\ 上式中H中元素h_{\theta }(x^{(i)})为\beta 中元素\beta (x^{(i)})逐个带入sigmoid函数得到 \end{gathered}$$
 \  

• 将(17)式子变化并同时带入上述矩阵化后的子式
   \  

$$\begin{gathered} \nabla f=\left[\begin{array}{c}\frac{\partial f}{\partial {\theta }_1}\\ \frac{\partial f}{\partial {\theta }_2}\\ \frac{\partial f}{\partial {\theta }_3}\\ .\\ .\\ \frac{\partial f}{\partial {\theta }_n}\end{array}\right]=\left[\begin{array}{c}{x}_1^1[y^1-h_{\theta }(x^1)]+x_1^2[y^2-h_{\theta }(x^2)]+...+x_1^2[y^n-h_{\theta }(x^n)]\\ x_2^1[y^1-h_{\theta }(x^1)]+x_2^2[y^2-h_{\theta }(x^2)]+...+x_2^2[y^n-h_{\theta }(x^n)]\\ x_3^1[y^1-h_{\theta }(x^1)]+x_3^2[y^2-h_{\theta }(x^2)]+...+x_3^2[y^n-h_{\theta }(x^n)]\\ .\\ .\\ x_n^1[y^1-h_{\theta }(x^1)]+x_n^2[y^2-h_{\theta }(x^2)]+...+x_n^2[y^n-h_{\theta }(x^n)]\end{array}\right]= \\ {X}^T(Y-H)\text{\hspace{1em}(19)} \end{gathered}$$

 \  
 \  

$$最后(18)可以写成这样子$$

$$\Theta =\Theta -\nabla f\text{\hspace{1em}(20)}$$

$$重复(19)-(20)步足够的次数即可得到最优的({\theta }_1,{\theta }_2,...{\theta }_n)使得代价函数(12)取最小值$$