第二课:Improving Deep Neural Networks 第一周：编程作业: Gradient Checking

第二门课 改善深层神经网络：超参数调试、正则化以及优化(Improving Deep Neural Networks:Hyperparameter tuning, Regularization and Optimization)

第一周：编程作业: Gradient Checking

本周课程笔记见：第一周：深度学习的实践层面(Practical aspects of Deep Learning)

Gradient Checking

Welcome to the final assignment for this week! In this assignment you will learn to implement and use gradient checking.

You are part of a team working to make mobile payments available globally, and are asked to build a deep learning model to detect fraud–whenever someone makes a payment, you want to see if the payment might be fraudulent, such as if the user’s account has been taken over by a hacker.

But backpropagation is quite challenging to implement, and sometimes has bugs. Because this is a mission-critical application, your company’s CEO wants to be really certain that your implementation of backpropagation is correct. Your CEO says, “Give me a proof that your backpropagation is actually working!” To give this reassurance, you are going to use “gradient checking”.

Let’s do it!

# Packages
import numpy as np
from testCases import *
from gc_utils import sigmoid, relu, dictionary_to_vector, vector_to_dictionary, gradients_to_vector

1) How does gradient checking work?

Backpropagation computes the gradients $\frac{\partial J}{\partial \theta }$, where $\theta$ denotes the parameters of the model. $J$ is computed using forward propagation and your loss function.

Because forward propagation is relatively easy to implement, you’re confident you got that right, and so you’re almost 100% sure that you’re computing the cost $J$ correctly. Thus, you can use your code for computing $J$ to verify the code for computing $\frac{\partial J}{\partial \theta }$.

Let’s look back at the definition of a derivative (or gradient):
$$\frac{\partial J}{\partial \theta }=\underset{\epsilon \to 0}{\lim }\frac{J(\theta +\epsilon )-J(\theta -\epsilon )}{2\epsilon }\text{\hspace{1em}(1)}$$

If you’re not familiar with the “$\underset{\epsilon \to 0}{\lim }$” notation, it’s just a way of saying “when $\epsilon$ is really really small.”

We know the following:

• $\frac{\partial J}{\partial \theta }$ is what you want to make sure you’re computing correctly.
• You can compute $J(\theta +\epsilon )$ and $J(\theta -\epsilon )$ (in the case that $\theta$ is a real number), since you’re confident your implementation for $J$ is correct.

Lets use equation (1) and a small value for $\epsilon$ to convince your CEO that your code for computing $\frac{\partial J}{\partial \theta }$ is correct!

2) 1-dimensional gradient checking

Consider a 1D linear function $J(\theta )=\theta x$. The model contains only a single real-valued parameter $\theta$, and takes $x$ as input.

You will implement code to compute $J(.)$ and its derivative $\frac{\partial J}{\partial \theta }$. You will then use gradient checking to make sure your derivative computation for $J$ is correct.

Figure 1 : 1D linear model

The diagram above shows the key computation steps: First start with $x$, then evaluate the function J ( x ) J(x)