二分类——识别猫

二分类 —— 逻辑回归

符号说明

n：输入特征向量的维度

$(x,y)$ $x\in {\mathbb{R}}^{n_x}$，y∈{0,1}y \in \{0,1\}y∈{0,1}：一个单独的样本

m：训练集中的训练样本数量，强调是训练集的时候用 $m_{train}$ 表示训练样本，用 $m_{test}$ 表示测试样本

{(x(1),y(1)),(x(2),y(2)),…,(x(m),y(m))}\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),\dots,(x^{(m)},y^{(m)})\}{(x(1),y(1)),(x(2),y(2)),…,(x(m),y(m))}：训练集

训练集：$X=\left[\begin{array}{cccc}⋮& ⋮& & ⋮\\ x^{(1)}& x^{(2)}& \cdots & x^{(m)}\\ ⋮& ⋮& & ⋮\end{array}\right]$ $X\in {\mathbb{R}}^{n_x\times m}$ n行m列

输出：$Y=\left[\begin{array}{cccc}{y}^{(1)}& y^{(2)}& \cdots & y^{(m)}\end{array}\right]$ $Y\in {\mathbb{R}}^{1\times m}$ 1行m列

逻辑回归模型

给定 $x$，$x\in {\mathbb{R}}^{n_x}$，想到得到 $\hat{y}$，是一个估计的概率：$\hat{y}=P(y=1|x)$ $0\le \hat{y}\le 1$ cans

参数：$\omega \in {\mathbb{R}}^{n_x}$，$b\in \mathbb{R}$

输出：$\hat{y}=\sigma ({\omega }^{T}x+b)=\sigma (z)=\frac{1}{1+e^{-z}}$ z 正向很大时$\sigma (z)\approx 1$，z 负向很大时$\sigma (z)\approx 0$，

• 损失函数：衡量单一训练样本的效果

  $L(\hat{y},y)=-(y\log \hat{y}+(1-y)\log (1-\hat{y}))$

  if y=1： $L(\hat{y},y)=-\log \hat{y}\leftarrow$ want $\log \hat{y}$ large，want $\hat{y}$ large
  if y=0： $L(\hat{y},y)=-\log （1-\hat{y}）\leftarrow$ want $\log （1-\hat{y}$ large，want $\hat{y}$ small

• 代价函数：衡量参数w和b在全部训练集上的效果
  $J(\omega ,b)=\frac{1}{m}{\sum }_{i=1}^{m}L({\hat{y}}^{(i)},y^{(i)})=-\frac{1}{m}{\sum }_{i=1}^m{y}^{(i)}\log {\hat{y}}^{(i)}+(1-y^{(i)})\log (1-{\hat{y}}^{(i)})$

使用这样的损失函数可以使得代价函数是一个凸函数

梯度下降

Repeat{
$\omega :=\omega -\alpha \frac{dJ(\omega ,b)}{dw}$
$b:=b-\alpha \frac{dJ(\omega ,b)}{db}$
}

• $\alpha$ 学习率
• 大于1时用偏导，但在编程中都记为 $\omega :=\omega -\alpha dw$

计算图

正向：从左向右计算代价函数，需要优化的函数

反向：从右向左计算导数

逻辑回归的梯度下降

单一样本：真值标签值$y$，两个特征值 $x_1$，$x_2$，输入参数${\omega }_1$，${\omega }_2$，$b$
正向预测：
$z={\omega }^{T}x+b$
$\hat{y}=a=\sigma (z)$
$L(a,y)=-(y\log a+(1-y)\log (1-a))$

反向求导：
$\text{da}\to \frac{dL(a,y)}{da}=-\frac{y}{a}+\frac{1-y}{1-a}$
$\text{dz}\to \frac{dL(a,y)}{dz}=\frac{dL(a,y)}{da}\frac{da}{dz}=a-y\leftarrow \frac{da}{dz}=\frac{e^{-z}}{(1+e^{-z}{)}^2}=\frac{1}{(1+e^{-z})}(1-\frac{1}{(1+e^{-z})})=a(1-a)$
$\text{dw1}\to \frac{dL(a,y)}{d{\omega }_1}=x_{1}dz$
$\text{dw2}\to \frac{dL(a,y)}{d{\omega }_2}=x_{2}dz$
$\text{db}\to \frac{dL(a,y)}{db}=dz$

m个样本：真值标签值 $y$，两个特征值 $x_1$，$x_2$，输入参数${\omega }_1$，${\omega }_2$，$b$

                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         $J=0,d{\omega }_1=0,d{\omega }_2=0,db=0$
                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         $\text{for i=1 to m:}$
$$\begin{array}{rl}& z^{(i)}={\omega }^T{x}^{(i)}+b\\ & a^{(i)}=\sigma (z{)}^{(i)}\\ & J+=-[y^{(i)}\log a^{(i)}+(1-y^{(i)})\log (1-a^{(i)})]\\ & d{z}^{(i)}=a^{(i)}-y^{(i)}\\ & d{\omega }_1^{(i)}+=x_1^{(i)}d{z}^{(i)}\\ & d{\omega }_2^{(i)}+=x_2^{(i)}d{z}^{(i)}\\ & db+=d{z}^{(i)}\end{array}$$                                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         $J=J/m,d{\omega }_1=d{\omega }_1/m,d{\omega }_2=d{\omega }_2/m,db=db/m$
                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         ${\omega }_1:={\omega }_1-\alpha d{\omega }_1$
                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         ${\omega }_2:={\omega }_2-\alpha d{\omega }_2$
                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         ​$b:=b-\alpha db$

向量化

$X=\left[\begin{array}{cccc}⋮& ⋮& & ⋮\\ x^{(1)}& x^{(2)}& \cdots & x^{(m)}\\ ⋮& ⋮& & ⋮\end{array}\right]$ $X\in {\mathbb{R}}^{n_x\times m}$ n行m列

$Z=\left[\begin{array}{c}{z}^{(1)},z^{(2)},\cdots ,z^{(i)}\end{array}\right]$

$A=\left[\begin{array}{c}{a}^{(1)},a^{(2)},\cdots ,a^{(i)}\end{array}\right]=\sigma (Z)$

                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~                                                         $\text{for inter in rang(1000):}$
$$\begin{array}{rl}& Z={\omega }^{T}X+b=np.dot(\omega .T,X)+b\\ & A=\sigma (Z)\\ & dZ=A-Y\\ & d\omega =\frac{1}{m}Xd{Z}^T\\ & db=\frac{1}{m}np.sum(dZ)\\ & \omega :=\omega -\alpha d\omega \\ & b:=b-\alpha db\end{array}$$

a logistic regression classifier to recognize cats

import numpy as np
import matplotlib.pyplot as plt
import h5py
import scipy
from PIL import Image
from scipy import ndimage
from lr_utils import load_dataset

%matplotlib inline

def load_dataset():
    train_dataset = h5py.File('datasets/train_catvnoncat.h5', "r")
    train_set_x_orig = np.array(train_dataset["train_set_x"][:]) # your train set features
    train_set_y_orig = np.array(train_dataset["train_set_y"][:]) # your train set labels

    test_dataset = h5py.File('datasets/test_catvnoncat.h5', "r")
    test_set_x_orig = np.array(test_dataset["test_set_x"][:]) # your test set features
    test_set_y_orig = np.array(test_dataset["test_set_y"][:]) # your test set labels

    classes = np.array(test_dataset["list_classes"][:]) # the list of classes
    
    train_set_y_orig = train_set_y_orig.reshape((1, train_set_y_orig.shape[0]))
    test_set_y_orig = test_set_y_orig.reshape((1, test_set_y_orig.shape[0]))
    
    return train_set_x_orig, train_set_y_orig, test_set_x_orig, test_set_y_orig, classes


# Loading the data (cat/non-cat)
train_set_x_orig, train_set_y, test_set_x_orig, test_set_y, classes = load_dataset()
#Figure out the dimensions and shapes of the problem (m_train, m_test, num_px, ...)
m_train = train_set_x_orig.shape[0]
m_test = test_set_x_orig.shape[0]
num_px = train_set_x_orig.shape[1]
print ("Number of training examples: m_train = " + str(m_train))
print ("Number of testing examples: m_test = " + str(m_test))
print ("Height/Width of each image: num_px = " + str(num_px))
print ("Each image is of size: (" + str(num_px) + ", " + str(num_px) + ", 3)")
print ("train_set_x shape: " + str(train_set_x_orig.shape))
print ("train_set_y shape: " + str(train_set_y.shape))
print ("test_set_x shape: " + str(test_set_x_orig.shape))
print ("test_set_y shape: " + str(test_set_y.shape))
#Reshape the datasets such that each example is now a vector of size (num_px * num_px * 3, 1)
train_set_x_flatten = train_set_x_orig.reshape(train_set_x_orig.shape[0],-1).T
test_set_x_flatten = test_set_x_orig.reshape(test_set_x_orig.shape[0],-1).T
#"Standardize" the data
train_set_x = train_set_x_flatten/255.
test_set_x = test_set_x_flatten/255.

def sigmoid(z):
    s = 1/(1+np.exp(-z))
    return s

def initialize_with_zeros(dim):
    w = np.zeros((dim,1))
    b = 0   
    assert(w.shape == (dim, 1))
    assert(isinstance(b, float) or isinstance(b, int))
    return w, b

def propagate(w, b, X, Y):
    m = X.shape[1]    
    # FORWARD PROPAGATION (FROM X TO COST)
    A = sigmoid(np.dot(w.T,X)+b)                        # compute activation
    cost = -1/m*np.sum(Y*np.log(A)+(1-Y)*np.log(1-A))   # compute cost
    # BACKWARD PROPAGATION (TO FIND GRAD)
    dw = 1/m*np.dot(X,(A-Y).T)
    db = 1/m*np.sum(A-Y)
    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    cost = np.squeeze(cost)
    assert(cost.shape == ())
    grads = {"dw": dw,
             "db": db}
    
    return grads, cost

def optimize(w, b, X, Y, num_iterations, learning_rate, print_cost = False):
    costs = []
    for i in range(num_iterations):
        # Cost and gradient calculation 
        grads, cost = propagate(w,b,X,Y)      
        # Retrieve derivatives from grads
        dw = grads["dw"]
        db = grads["db"]      
        # update rule
        w = w-learning_rate*dw
        b = b-learning_rate*db
        # Record the costs
        if i % 100 == 0:
            costs.append(cost)
        # Print the cost every 100 training examples
        if print_cost and i % 100 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))  
    params = {"w": w,
              "b": b} 
    grads = {"dw": dw,
             "db": db}
    return params, grads, costs

def predict(w, b, X):
    m = X.shape[1]
    Y_prediction = np.zeros((1,m))
    w = w.reshape(X.shape[0], 1)
    # Compute vector "A" predicting the probabilities of a cat being present in the picture
    A = sigmoid(np.dot(w.T,X)+b)
    for i in range(A.shape[1]):    
        # Convert probabilities A[0,i] to actual predictions p[0,i]
        if A[0,i]<=0.5:
            Y_prediction[0,i]=0
        else:
            Y_prediction[0,i]=1 
    assert(Y_prediction.shape == (1, m))  
    return Y_prediction

def model(X_train, Y_train, X_test, Y_test, num_iterations = 2000, learning_rate = 0.5, print_cost = False): 
    # initialize parameters with zeros 
    w, b = initialize_with_zeros(X_train.shape[0])
    # Gradient descent 
    parameters, grads, costs = optimize(w, b, X_train, Y_train, num_iterations, learning_rate, print_cost)
    # Retrieve parameters w and b from dictionary "parameters"
    w = parameters["w"]
    b = parameters["b"]
    # Predict test/train set examples 
    Y_prediction_test = predict(w,b,X_test)
    Y_prediction_train = predict(w,b,X_train)
    # Print train/test Errors
    print("train accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100))
    print("test accuracy: {} %".format(100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100))
    d = {"costs": costs,
         "Y_prediction_test": Y_prediction_test, 
         "Y_prediction_train" : Y_prediction_train, 
         "w" : w, 
         "b" : b,
         "learning_rate" : learning_rate,
         "num_iterations": num_iterations}
    return d

d = model(train_set_x, train_set_y, test_set_x, test_set_y, num_iterations = 2000, learning_rate = 0.005, print_cost = True)

# Plot learning curve (with costs)
plt.figure(1)
costs = np.squeeze(d['costs'])
plt.plot(costs)
plt.ylabel('cost')
plt.xlabel('iterations (per hundreds)')
plt.title("Learning rate =" + str(d["learning_rate"]))
plt.show()

index = 2
plt.figure(2)
plt.imshow(test_set_x[:,index].reshape((num_px, num_px, 3)))
print ("y = " + str(test_set_y[0,index]) + ", you predicted that it is a \"" + classes[int(d["Y_prediction_test"][0,index])].decode("utf-8") +  "\" picture.")

# change this to the name of your image file 
my_image = "my_image2.jpg"
# We preprocess the image to fit your algorithm.
fname = "images/" + my_image
image = np.array(ndimage.imread(fname, flatten=False))
my_image = scipy.misc.imresize(image, size=(num_px,num_px)).reshape((1, num_px*num_px*3)).T
my_predicted_image = predict(d["w"], d["b"], my_image)
plt.figure(3)
plt.imshow(image)
print("y = " + str(np.squeeze(my_predicted_image)) + ", your algorithm predicts a \"" + classes[int(np.squeeze(my_predicted_image)),].decode("utf-8") +  "\" picture.")

Number of training examples: m_train = 209
Number of testing examples: m_test = 50
Height/Width of each image: num_px = 64
Each image is of size: (64, 64, 3)
train_set_x shape: (209, 64, 64, 3)
train_set_y shape: (1, 209)
test_set_x shape: (50, 64, 64, 3)
test_set_y shape: (1, 50)
Cost after iteration 0: 0.693147
Cost after iteration 100: 0.584508
Cost after iteration 200: 0.466949
Cost after iteration 300: 0.376007
Cost after iteration 400: 0.331463
Cost after iteration 500: 0.303273
Cost after iteration 600: 0.279880
Cost after iteration 700: 0.260042
Cost after iteration 800: 0.242941
Cost after iteration 900: 0.228004
Cost after iteration 1000: 0.214820
Cost after iteration 1100: 0.203078
Cost after iteration 1200: 0.192544
Cost after iteration 1300: 0.183033
Cost after iteration 1400: 0.174399
Cost after iteration 1500: 0.166521
Cost after iteration 1600: 0.159305
Cost after iteration 1700: 0.152667
Cost after iteration 1800: 0.146542
Cost after iteration 1900: 0.140872
train accuracy: 99.04306220095694 %
test accuracy: 70.0 %