神经网络与深度学习_神经网络基础_第二周笔记

Supervised learning

Application:

• Standard NN
  • real estate
  • Online advertising
• CNN
  • Photo tagging
• RNN
  • Speech recognition
  • Machine translation
• Custom/hybrid RNNs
  • Autonomous driving

---

Notation

(x,y),x∈Rnx,y∈{0,1}(x,y), x∈R^{n_x} ,y∈\{0,1\}(x,y),x∈Rnx​,y∈{0,1}
m training example {$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(m)},y^{(m)})$}
$$X=\left[\begin{array}{cccccc}.& .& .& .& .& -\\ .& .& .& .& .& |\\ x^{(1)}& x^{(2)}& ...& ...& x^{(m)}& n_x\\ .& .& .& .& .& |\\ .& .& .& .& .& -\\ <-& & m& & ->& \end{array}\right]$$
$X\in R^{n_x\ast m},Y=[y^{(1)},y^{(2)},..,y^{(m)}],Y\in R^{1\ast m}$

x.shape(nx,m) y.shape=(1,m)

---

Logistic Regression

$x\in R^{n_x},want:\hat{y}=P(y=1|x),so0\le \hat{y}\le 1$
$parameters:w\in R^{n_x},b\in R$
$Output:\hat{y}=\sigma (w^{T}x+b)$
$\sigma$ is activation function $\sigma (z)=\frac{1}{1+e^{(-z)}}$

---

Logistic Regression Cost Function

${\hat{y}}^{(i)}=\sigma (w^T{x}^{(i)}+b)$

given {$(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(m)},y^{(m)})$} want ${\hat{y}}^{(i)}\approx y^{(i)}$

Measure single training sample

usually can use

$L(\hat{y},y)=\frac{1}{2}(\hat{y}-y{)}^2$

to measure the gap but later gradient descent may not work well because it’s non-convex function

$\hat{y}=\sigma (w^{T}x+b),where:\sigma (z)=\frac{1}{1+e^{-z}},interpret:\hat{y}=P(y=1|x)$
IF $y=1:P(y|x)=\hat{y}$
IF $y=0:P(y|x)=1-\hat{y}$

combine the function above
$$P(y|x)={\hat{y}}^y(1-\hat{y}{)}^{1-y}$$and the $\log$ function is a strictly monotonically increasing function
$$\log P(y|x)=y\log \hat{y}+(1-y)\log (1-\hat{y})$$then add negative sign because we want the minimum cost, so

$$\begin{array}{rl}-\log P(y|x)=& -[y\log \hat{y}+(1-y)\log (1-\hat{y})]\\ L(\hat{y},y)=& -[y\log \hat{y}+(1-y)\log (1-\hat{y})]\end{array}$$

this is the cost function with single example

Cost function in m training set

under IID
$$P(labels-in-training-set)=\prod _{i=1}^{m}P(y^{(i)}|x^{(i)})$$ to maximizing the training set chance as same as maximizing the $\log$ fun

$$\begin{array}{rl}\log P(labels-in-training-set)=& \log \prod _{i=1}^{m}P(y^{(i)}|x^{(i)})\\ =& \sum _{i=1}^m\log P(y^{(i)}|x^{(i)})\\ =& \sum _{i=1}^m-L({\hat{y}}^{(i)},y^{(i)})\\ =& \frac{1}{m}\sum _{i=1}^m-L({\hat{y}}^{(i)},y^{(i)})\end{array}$$

add $\frac{1}{m}$ scaling factor to for better scale
so the overall cost function

$$J(w,b)=\frac{1}{m}\sum _{i=1}^{m}L({\hat{y}}^{(i)},y^{(i)})$$

remove the nagative for minimize the cost function
$J(w,b)$ is convex func, that is the particular reason to be chosen for cost function

---

Gradient Descent

Repeat:{
$w:=w-\alpha \frac{dJ(w,b)}{dw}$
$b:=b-\alpha \frac{dJ(w,b)}{db}$
}
$\alpha$ is the learning rate

Computation Graph

from right to left

$\frac{dL}{da}=-\frac{y}{a}+\frac{1-y}{1-a}$

$\frac{dL}{dz}=\frac{dL}{da}\cdot \frac{da}{dz}=(-\frac{y}{a}+\frac{1-y}{1-a})(a(1-a))=a-y=dz$

$\frac{dL}{d{w}_1}=\frac{dL}{dz}\cdot \frac{dz}{d{w}_1}=dz\cdot x_1=d{w}_1$

$\frac{dL}{d{w}_2}=\frac{dL}{dz}\cdot \frac{dz}{d{w}_2}=dz\cdot x_2=d{w}_2$

$\frac{dL}{db}=\frac{dL}{dz}\cdot \frac{dz}{db}=dz=db$

so in single example :

${\omega }_1:={\omega }_1-\alpha dz\cdot x_1$
${\omega }_2:={\omega }_2-\alpha dz\cdot x_2$
$b:=b-\alpha dz$

---

Gradient descent in m example

$$J(w,b)=\frac{1}{m}\sum _{i=1}^{m}L(a^{(i)},y^{(i)}),a^{(i)}={\hat{y}}^{(i)}=\sigma (z^{(i)})=\sigma ({\omega }^T{x}^{(i)}+b)$$

$$\begin{array}{rl}\frac{dJ(w_1,b)}{d{w}_1}=& \frac{1}{m}\sum _{i=1}^m\frac{dL(a^{(i)},y^{(i)})}{d{w}_1}\\ =& \frac{1}{m}\sum _{i=1}^{m}d{w}_1^{(i)}--using(x_1^{(i)},y^{(i)})\end{array}$$

J=0 dw1=0 dw2=0 b=0
for i=1 to m
    z[i]=w_T*x[i]+b
    a[i]=sigma(z[i])
    J[i]+=-[y[i]*log(a[i])+(1-y[i])*log(1-a[i])]
    dz[i]=a[i]-y[i]
    for w in wm
       dw[1] += dz[i]*x1[i]
       dw[2] +=dz[i]*x2[i]
       ...
       db +=dz[i]
     end
 end
 dw1=dw1/m dw2=dw2/m db=db/m J=J/m

There are 2 loops ,less efficiency $⟶$ Vectorization

---

Vectorization

$Z={\omega }^{T}x+b$

• non-vectorization

       z=0
       for i in range(nx):
           z += w[i]*x[i]
       z = z+b

• vectorization

  $$\omega =\left[\begin{array}{c}.\\ .\\ {\omega }^{(i)}\\ .\\ .\end{array}\right]X=\left[\begin{array}{c}.\\ .\\ x^{(i)}\\ .\\ .\end{array}\right]\omega \in R^{n_x},x\in R^{n_x}$$

dw=np.zeros(n_x,1) x.shape(n_x,1)

so in code

J=0  b=0
dw=np.zeros(n_x,1)
for i=1 to m //one loop for x
    z[i]=w_T*x[i]+b
    a[i]=sigma(z[i])
    J[i]+=-[y[i]* log(a[i])+(1-y[i])* log(1-a[i])]
    dz[i]=a[i]-y[i]
    dw += dw[i]* x[i]
    db +=dz[i]
 end
 dw=dw/m db=db/m J=J/m

Vectoring Logistic Regression

$$\begin{gathered} z^{(1)}={\omega }^T{x}^{(1)}+b,a^{(1)}=\sigma (z^{(1)}) \\ {z}^{(2)}={\omega }^T{x}^{(2)}+b,a^{(2)}=\sigma (z^{(2)}) \\ ... \end{gathered}$$

$$X=\left[\begin{array}{ccccc}.& .& .& .& .\\ .& .& .& .& .\\ x^{(1)}& x^{(2)}& ...& ...& x^{(m)}\\ .& .& .& .& .\\ .& .& .& .& .\end{array}\right]$$

$$\begin{array}{rl}Z=& [z^{(1)},z^{(2)},z^{(3)},...,z^{(m)}]\\ =& {\omega }^{T}X+[b,b,...,b]\\ =& [{\omega }^T{x}^{(1)}+b,{\omega }^T{x}^{(2)}+b,...,{\omega }^T{x}^{(m)}+b]\end{array}$$

Z =np.dot(w.T,x)+b . //python Broadcasting as a vector

Vecrotring Logistic Regression Gradient Descent

$$\begin{gathered} d{z}^{(i)}=a^{(i)}-y^{(i)}, \\ dz=[d{z}^{(1)},d{z}^{(2)},...,d{z}^{(m)}], \\ A=[a^{(1)},a^{(2)},..,a^{(m)}], \\ Y=[y^{(1)},y^{(2)},...,y^{(m)}], \\ dz=A-Y \\ db=\frac{1}{m}{\sum }_{i=1}^{m}d{z}^{(i)} \end{gathered}$$

db=np.sum(dz)/m

$d\omega =\frac{1}{m}\cdot X\cdot d{z}^T$
$d\omega =\frac{1}{m}\left[\begin{array}{ccccc}.& .& .& .& .\\ .& .& .& .& .\\ x^{(1)}& x^{(2)}& ...& ...& x^{(m)}\\ .& .& .& .& .\\ .& .& .& .& .\end{array}\right]\cdot \left[\begin{array}{c}{d}^{(1)}\\ d^{(2)}\\ .\\ .\\ d^{(m)}\end{array}\right]=\frac{1}{m}[x^{(1)}d{z}^{(1)}+x^{(2)}d{z}^{(2)}...+x^{(m)}d{z}^{(m)}]$

for iter in range(2000):
 z=np.dot(W.T,X)+b
 A=sigmoid(z)
 dz=A-Y
 dw=np.dot(X,dz.T)/m
 db=np.sum(dz)/m
 W=W-alpha * dw
 b=b-alpha * db