学习笔记 - GreedyAI - DeepLearningCV - Lesson1 Introduction

第3章 逻辑回归

任务学习12 二元分类问题

任务学习13 逻辑函数

$f(x)=\frac{1}{1+e^{-x}}$

$\frac{1}{1+e^{\infty }}=\frac{1}{1+\infty }=0$

$\frac{1}{1+e^{-\infty }}=\frac{1}{1-\infty }=1$

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

def sigmoid(x):
    return 1 / (1 + np.exp( - x ))

x = np.linspace(-6, 6, 100)
y = sigmoid(x)
mark = 0.5 * np.ones(x.shape)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(x, y)
ax.plot(x, mark, ":")
ax.set_xlabel("$x$")
ax.set_ylabel("$f(x)$")
ax.grid()
plt.show()

任务学习14 指数与对数 、逻辑回归

• 指数与对数

def exp(x):
    return np.exp(x)

def ln(x):
    return np.log(x)

def lin(x):
    return x

x = np.linspace(-4, 4, 100)
y_exp = exp(x)
y_ln = ln(x[np.nonzero(x > 0)])
y_lin = lin(x)

fig = plt.figure(figsize = (5, 5))
ax = fig.add_subplot(111)
ax.plot(x, y_exp, label="$y = e^{x}$")
ax.plot(x[np.nonzero(x > 0)], y_ln, label="$y = ln(x)$")
ax.plot(x, y_lin, label="$y = x$")
ax.set_xlabel("$x$")
ax.set_ylabel("$f(x)$")
ax.set_ylim(-4, 4)
ax.grid()
ax.legend()
plt.show()

• 逻辑回归

解决二元（0、1）分类问题

$P(y=1|\mathbf{x};\theta )=f(\mathbf{x};\theta )=\frac{1}{1+e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}$

${\theta }^{\mathrm{T}}\mathbf{x}={\theta }_0+{\theta }_1{x}_1+{\theta }_2{x}_2+\cdots$

$\theta =\left[{\theta }_0,{\theta }_1,{\theta }_2,\cdots \right]$

$\mathbf{x}=\left[1,x_1,x_2,\cdots \right]$

$P(y=1|\mathbf{x})>0.5$，推理为1；否则推理为0。

• 逻辑回归知识点

类别1的概率：$P=\frac{1}{1+e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}$

类别0的概率：$1-P=\frac{e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}{1+e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}=\frac{1}{1+e^{{\theta }^{\mathrm{T}}\mathbf{x}}}$

类别1与0概率的比值：$\frac{P}{1-P}=e^{{\theta }^{\mathrm{T}}\mathbf{x}}$

类别1与0概率比值的自然对数：$\ln \frac{P}{1-P}={\theta }^{\mathrm{T}}\mathbf{x}$

任务学习15 逻辑回归示例

年龄（$x_1$）	年收入（$x_2$）（万元）	是否买车（1：是；2：否）
20	3	0
23	7	1
31	10	1
42	13	1
50	7	0
60	5	0
—	—	—
28	8	？

from sklearn import linear_model

X = [[20, 3],
     [23, 7],
     [31, 10],
     [42, 13],
     [50, 7],
     [60, 5]]

y = [0,
     1,
     1,
     1,
     0,
     0]

lr = linear_model.LogisticRegression()
lr.fit(X, y)

testX = [[28, 8]]

label = lr.predict(testX)
print("predicted label = {}".format(label))

prob = lr.predict_proba(testX)
print("probability = {}".format(prob))

print("theta_0 = {0[0]}, theta_1 = {1[0][0]}, theta_0 = {1[0][1]}".format(lr.intercept_, lr.coef_))

predicted label = [1]
probability = [[0.14694811 0.85305189]]
theta_0 = -0.04131837596993478, theta_1 = -0.1973000136829152, theta_0 = 0.915557452347983

任务学习16 损失函数

类别1概率：

$P(y=1|\mathbf{x};\theta )=f(\mathbf{x};\theta )=\frac{1}{1+e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}$

损失函数：

$J(\theta )=-{\sum }_{i=1}^N\left[y^{(i)}\ln P(Y=1|\mathbf{X}={\mathbf{x}}^{(i)};\theta )+\left(1-y^{(i)}\right)\ln \left(1-P(Y=1|\mathbf{X}={\mathbf{x}}^{(i)};\theta )\right)\right]$

损失函数梯度：

$\begin{array}{rl}{\nabla }_{\theta }J(\theta )=& \sum _{i=1}^N\left(P(Y=1|\mathbf{X}={\mathbf{x}}^{(i)};\theta )-y^{(i)}\right){\mathbf{x}}^{(i)}\\ =& \sum _{i=1}^N{\mathbf{x}}^{(i)}\left(f({\mathbf{x}}^{(i)};\theta )-y^{(i)}\right)\end{array}$

任务学习17 损失函数推演

1. 求导

$${\left(f(x)g(x)\right)}^{\prime }=f^{\prime }(x)g(x)+f(x)g^{\prime }(x)$$

2. 对数

$$\log (xy)=\log (x)+\log (y)$$

$${\log }^{\prime }(x)=\frac{1}{x}$$

3. 链式法则

$$\begin{array}{rl}z=& f(y)\\ y=& g(x)\\ \downarrow & \\ \frac{dz}{dx}=& \frac{dz}{dy}\frac{dy}{dx}\end{array}$$

4. sigmoid

$$\begin{array}{rl}f(x)=& \frac{1}{1+e^{-x}}\\ \downarrow & \\ f^{\prime }(x)=& (-1)\frac{e^{-x}(-1)}{{\left(1+e^{-x}\right)}^2}\\ =& \frac{e^{-x}}{1+e^{-x}}\frac{1}{1+e^{-x}}\\ =& f(x)\left(1-f(x)\right)\end{array}$$

$$\begin{array}{rl}f(z)=& \frac{1}{1+e^{-z}}\\ z=& \theta x\\ \downarrow & \\ \frac{df}{dx}=& f(z)\left(1-f(z)\right)\theta \end{array}$$

5. 损失函数

训练数据集{(xi,yi)}\{ \left( \mathbf{x}_i, y_i \right) \}{(xi​,yi​)}，i∈{1,2,⋯ ,N}i \in \{1, 2, \cdots, N \}i∈{1,2,⋯,N}，${\mathbf{x}}_i\in R^m$，yi∈{0,1}y_i \in \{ 0, 1 \}yi​∈{0,1}

逻辑函数表示给定样本${\mathbf{x}}_i$，分类器推理为$y_i=1$的概率：

$$\begin{array}{rl}{P}_i=& P\left(y_i=1|\theta :{\mathbf{x}}_i\right)\\ =& f({\theta }^{\mathrm{T}}{\mathbf{x}}_i)\end{array}$$

似然函数

$$\begin{array}{rl}L(\theta )=& \prod _{i|y_i=1}^N{P}_i\cdot \prod _{i|y_i=0}^N\left(1-P_i\right)\end{array}$$

目标是求使$L(\theta )$最大时的$\theta$：

$$\theta =\arg \underset{\theta }{\max }L(\theta )$$

对数似然函数

$$\begin{array}{rl}l(\theta )=\log L(\theta )=& \log \left[\sum _{i|y_i=1}^N{P}_i+\sum _{i|y_i=0}^N\left(1-P_i\right)\right]\\ =& \sum _{i|y_i=1}^N\log P_i+\sum _{i|y_i=0}^N\log \left(1-P_i\right)\\ =& \sum _{i=1}^N\left[y_i\log P_i+\left(1-y_i\right)\log \left(1-P_i\right)\right]\end{array}$$

$$\begin{array}{rl}\frac{dl(\theta )}{d\theta }=& \sum _{i=1}^N\left[y_i\frac{d\log P_i}{d\theta }+\left(1-y_i\right)\frac{d\log \left(1-P_i\right)}{d\theta }\right]\\ =& \sum _{i=1}^N\left[y_i\frac{P_i\left(1-P_i\right)}{P_i}{\mathbf{x}}_i+\left(1-y_i\right)\frac{(-1)P_i\left(1-P_i\right)}{1-P_i}{\mathbf{x}}_i\right]\\ =& \sum _{i=1}^N\left[y_i\left(1-P_i\right){\mathbf{x}}_i-\left(1-y_i\right)P_i{\mathbf{x}}_i\right]\\ =& \sum _{i=1}^N\left(y_i-P_i\right){\mathbf{x}}_i\end{array}$$

$l(\theta )=\log L(\theta )$是求$L(\theta )$的最大期望，定义损失函数为：

$$loss(\theta )=-l(\theta )$$

则：

$$\frac{dloss(\theta )}{d\theta }=\sum _{i=1}^N\left(P_i-y_i\right){\mathbf{x}}_i$$

任务学习18 梯度下降法

$$f(\mathbf{x};\theta )=\frac{1}{1+e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}$$

$$\theta =\theta -\alpha {\nabla }_{\theta }J(\theta )=\theta -\alpha \sum _{i=1}^N{\mathbf{x}}^{(i)}\left(f({\mathbf{x}}^{(i)};\theta )-y^{(i)}\right)$$

• 系数的意义

概率比值$odds=\frac{P}{1-P}=e^{{\theta }^{\mathrm{T}}\mathbf{x}}$

系数${\theta }_j$意味着：假设原始$odds={\lambda }_1$，若对应的特征$x_j$增加1，假设新的$odds={\lambda }_2$，则$\frac{{\lambda }_1}{{\lambda }_2}\equiv e^{{\theta }_j}$

theta_0 = lr.intercept_
theta_1 = lr.coef_[0][0]
theta_2 = lr.coef_[0][1]

print("theta_0 = {0[0]}, theta_1 = {1}, theta_2 = {2}".format(theta_0, theta_1, theta_2))

testX = [[28, 8]]
ratio = prob[0][1] / prob[0][0]

testX = [[28, 9]]
prob_new = lr.predict_proba(testX)
ratio_new = prob_new[0][1] / prob_new[0][0]

ratio_of_ratio = ratio_new / ratio
print("ratio of ratio = {0}".format(ratio_of_ratio))

import math
theta2_e = math.exp(theta_2)
print("theta2 e = {}".format(theta2_e))

theta_0 = -0.04131837596993478, theta_1 = -0.1973000136829152, theta_2 = 0.915557452347983
ratio of ratio = 2.4981674731438943
theta2 e = 2.4981674731438948

${\theta }_2=0.92$意味着，如果年收入增加1万，一个人买车和不买车的概率的比值与之前的比值相比较，增加了$e^{0.92}=2.5$倍。

${\theta }_1=-0.20$意味着，如果年龄增加1岁，一个人买车和不买车的概率的比值与之前的比值相比较，降低了$e^{-0.20}=0.82$倍。

任务学习19 应用

import pandas as pd
from sklearn import linear_model
from sklearn.feature_extraction.text import TfidfVectorizer

df = pd.read_csv("./data/SMSSpamCollection.csv", delimiter=',', header=None)
y, X_train = df[0], df[1]

vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(X_train)

lr = linear_model.LogisticRegression()
lr.fit(X, y)

testX = vectorizer.transform(["URGENT! Your mobile No. 1234 was awarded a Prize.",
                              "Hey honey, what's up?"])

predictions = lr.predict(testX)
print(predictions)

['spam' 'ham']

PS：损失函数$J(\theta )$对$\theta$的Hessian矩阵：

• 损失函数：

$J(\theta )=-{\sum }_{i=1}^N\left[y^{(i)}\ln f(\mathbf{x};\theta )+\left(1-y^{(i)}\right)\ln \left(1-f(\mathbf{x};\theta )\right)\right]$

其中，
$f(\mathbf{x};\theta )=\frac{1}{1+e^{-{\theta }^{\mathrm{T}}\mathbf{x}}}$，
$\mathbf{x}={\left[1,x_1,x_2,\cdots ,x_n\right]}^{\mathrm{T}}$，
$\theta ={\left[{\theta }_0,{\theta }_1,{\theta }_2,\cdots ,{\theta }_n\right]}^{\mathrm{T}}$

其中，${\mathbf{x}}^{(i)}$为表示第$i$条样本的列向量。

• 损失函数$J(\theta )$对$\theta$的梯度：

$\begin{array}{rl}{\nabla }_{\theta }J(\theta )=& \sum _{i=1}^N{\mathbf{x}}^{(i)}\left(f({\mathbf{x}}^{(i)};\theta )-y^{(i)}\right)\end{array}$

• 损失函数$J(\theta )$对$\theta$的Hessian矩阵：

易知，$J(\theta )$对${\theta }_p$的一阶偏导数为：

$\begin{array}{rl}\frac{\partial J(\theta )}{\partial {\theta }_p}=& \sum _{i=1}^N{x}_p^{(i)}\left(f({\mathbf{x}}^{(i)};\theta )-y^{(i)}\right)\end{array}$

$J(\theta )$对${\theta }_p$和${\theta }_q$的二阶偏导数为：

$\begin{array}{rl}\frac{{\partial }^{2}J(\theta )}{\partial {\theta }_p\partial {\theta }_q}=& \sum _{i=1}^N{x}_p^{(i)}\frac{\partial f({\mathbf{x}}^{(i)};\theta )}{\partial {\theta }_q}\\ =& \sum _{i=1}^N{x}_p^{(i)}f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right)x_q^{(i)}\\ =& \sum _{i=1}^{N}f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right)x_p^{(i)}{x}_q^{(i)}\end{array}$

注意：$f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right)$为标量，且大于零。

$\begin{array}{rl}H\left(J(\theta )\right)=& \left[\begin{array}{cccc}\frac{{\partial }^{2}J(\theta )}{\partial {\theta }_1\partial {\theta }_1}& \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_1\partial {\theta }_2}& \cdots & \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_1\partial {\theta }_n}\\ \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_2\partial {\theta }_1}& \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_2\partial {\theta }_2}& \cdots & \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_2\partial {\theta }_n}\\ ⋮& ⋮& \ddots & ⋮\\ \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_n\partial {\theta }_1}& \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_n\partial {\theta }_2}& \cdots & \frac{{\partial }^{2}J(\theta )}{\partial {\theta }_n\partial {\theta }_n}\end{array}\right]\\ =& \sum _{i=1}^N\left(f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right)\left[\begin{array}{cccc}{x}_1^{(i)}{x}_1^{(i)}& x_1^{(i)}{x}_2^{(i)}& \cdots & x_1^{(i)}{x}_n^{(i)}\\ x_2^{(i)}{x}_1^{(i)}& x_2^{(i)}{x}_2^{(i)}& \cdots & x_2^{(i)}{x}_n^{(i)}\\ ⋮& ⋮& \ddots & ⋮\\ x_n^{(i)}{x}_1^{(i)}& x_n^{(i)}{x}_2^{(i)}& \cdots & x_n^{(i)}{x}_n^{(i)}\end{array}\right]\right)\\ =& \sum _{i=1}^{N}f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right){\mathbf{x}}^{(i)}({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}\end{array}$

• Hessian矩阵正定性分析

$$\begin{array}{rl}H\left(J(\theta )\right)=& \sum _{i=1}^{m}f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right){\mathbf{x}}^{(i)}({\mathbf{x}}^{(i)}{)}^{\mathrm{T}}\end{array}$$

（1）$f({\mathbf{x}}^{(i)};\theta )\left(1-f({\mathbf{x}}^{(i)};\theta )\right)>0$

（2）$H\left(J(\theta )\right)$在形式上类似于随机过程向量的自相关矩阵

当$m\gg 0$时，可得：

$$\mathrm{E}\left[x_j{x}_k\right]\approx \frac{1}{m}\sum _{i=1}^m{x}_j^{(i)}{x}_k^{(i)}$$

当${\mathbf{x}}^{(i)}$的各分量$x_j$相互独立时，可知：

$$\mathrm{E}\left[x_j{x}_k\right]\{\begin{array}{ll}=0,& \text{if}j̸=k\\ >0,& \text{if}j=k\end{array}$$

当$m\gg n$时，$\mathrm{E}\left[\mathbf{x}{\mathbf{x}}^{\text{T}}\right]$为满秩对角矩阵，且对角元素均大于零，$H\left(J(\theta )\right)$是正定的（positive definite)；否则$H\left(J(\theta )\right)$是半正定的（semi-positive definite)。

当$H\left(J(\theta )\right)$满足正定条件时（$m\gg n$），$J(\theta )$为凸优函数，有全局最优解，即批量梯度下降（batch gradient descent）能够保证$J(\theta )$收敛到全局最小值；当$H\left(J(\theta )\right)$满足半正定条件时（$m<n$），即小批量梯度下降（batch gradient descent）或随机梯度下降（stochastic gradient descent）可能使$J(\theta )$陷入局部最小值。