Neural Networks and Deep Learning 3

Ch03 Improving the way neural networks learn

在线书籍 http://neuralnetworksanddeeplearning.com/chap3.html

目录

Introducing the cross-entropy cost function

• Verify that ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$.
  证明： $\sigma (z)=\frac{1}{1+e^{-z}}$

  ${\sigma }^{\prime }(z)=-\frac{(1+e^{-z}{)}^{\prime }}{(1+e^{-z}{)}^2}=\frac{e^{-z}}{(1+e^{-z}{)}^2}=\frac{1}{1+e^{-z}}\cdot \frac{e^{-z}}{1+e^{-z}}$

  其中，$\frac{e^{-z}}{1+e^{-z}}=\frac{1+e^{-z}-1}{1+e^{-z}}=1-\frac{1}{1+e^{-z}}$

  由于 $\sigma (z)=\frac{1}{1+e^{-z}}$，

  所以 ${\sigma }^{\prime }(z)=\sigma (z)(1-\sigma (z))$.

• One gotcha with the cross-entropy is that it can be difficult at first to remember the respective roles of the $y$s and the $a$s. It’s easy to get confused about whether the right form is $-[y\ln a+(1-y)\ln (1-a)]$ or $[a\ln y+(1-a)\ln (1-y)]$. What happens to the second of these expressions when $y=0$ or $1$? Does this problem afflict the first expression? Why or why not?

  证明： 当 $C=-\frac{1}{n}{\sum }_x\left[y\ln a+(1-y)\ln (1-a)\right]$ （57）时，

  $\frac{\partial C}{\partial w_j}=\frac{1}{n}{\sum }_x{x}_j(\sigma (z)-y).$ (61)

  $\frac{\partial C}{\partial b}=\frac{1}{n}{\sum }_x(\sigma (z)-y).$ (62)

  无论 $y=1$ ，还是 $y=0$，计算过程都相对简单

  若 $C=-\frac{1}{n}{\sum }_x\left[a\ln y+(1-a)\ln (1-y)\right]$ ，

  则当 $y=1$ 时 $a\ln y=0$ ，$(1-a)\ln (1-y)$ 无意义

  或 $y=0$ 时 $(1-a)\ln (1-y)=0$，$a\ln y$ 无意义

  且 $\frac{\partial C}{\partial w_j}=\frac{1}{n}{\sum }_x{\sigma }^{\prime }(z)\ln \frac{y}{1-y}=\frac{1}{n}{\sum }_x{x}_j\sigma (z)(1-\sigma (z))\ln \frac{y}{1-y}$，其中无论 $y=0$ 还是 $y=1$，$\ln \frac{y}{1-y}$ 都无法计算；同理，$\frac{\partial C}{\partial b}=\frac{1}{n}{\sum }_x\sigma (z)(1-\sigma (z))\ln \frac{y}{1-y}$ 无法计算，因此不能用二式替换一式

• In the single-neuron discussion at the start of this section, I argued that the cross-entropy is small if $\sigma (z)\approx y$ for all training inputs. The argument relied on $y$ being equal to either $0$ or $1$. This is usually true in classification problems, but for other problems (e.g., regression problems) $y$ can sometimes take values intermediate between $0$ and $1$. Show that the cross-entropy is still minimized when $\sigma (z)=y$ for all training inputs. When this is the case the cross-entropy has the value:

  $$C=-\frac{1}{n}\sum _x\left[y\ln y+(1-y)\ln (1-y)\right](64)$$

  The quantity $-\left[y\ln y+(1-y)\ln (1-y)\right]$ is sometimes known as the binary entropy.

  证明： 在回归问题中，（交叉熵）损失函数 $C=-\frac{1}{n}{\sum }_x\left[y\ln a+(1-y)\ln (1-a)\right]$ （57），
  由于 $a\in (0,1)$ 所以 $\ln a,\ln (1-a)<0$ 且 $y\in (0,1)$，所以$C\ge 0$，且当 $a\to 0$ 时，$C\to \infty$，所以 $C$ 没有最大值。

  $\frac{\partial C}{\partial a}=-\frac{1}{n}{\sum }_x\left[\frac{y}{a}-\frac{1-y}{1-a}\right]=\frac{1}{n}{\sum }_x\frac{a-y}{a(1-a)}$

  由于 $a$ 取值范围为 $(0,1)$，所以 $a(1-a)>0$，当 $a>y$ 时， $\frac{\partial C}{\partial a}>0$ ，$C$ 关于$a$ 单调递增；当 $a<y$ 时，$\frac{\partial C}{\partial a}<0$，$C$ 关于$a$ 单调递减；因此当 $a=y$时取最小值。所以在回归问题中，也是最小化优化问题。

• Many-layer multi-neuron networks

  In the notation introduced in the last chapter, show that for the quadratic cost the partial derivative with respect to weights in the output layer is

  $$\frac{\partial C}{\partial w_{jk}^L}=\frac{1}{n}\sum _x{a}_k^{L-1}(a_j^L-y_j){\sigma }^{\prime }(z_j^L).(65)$$

  The term ${\sigma }^{\prime }(z_j^L)$ causes a learning slowdown whenever an output neuron saturates on the wrong value. Show that for the cross-entropy cost the output error ${\delta }^L$ for a single training example $x$ is given by
  $${\delta }^L=a^L-y.(66)$$

  Use this expression to show that the partial derivative with respect to the weights in the output layer is given by
  $$\frac{\partial C}{\partial w_{jk}^L}=\frac{1}{n}\sum _x{a}_k^{L-1}(a_j^L-y_j).(67)$$
  The ${\sigma }^{\prime }(z_j^L)$ term has vanished, and so the cross-entropy avoids the problem of learning slowdown, not just when used with a single neuron, as we saw earlier, but also in many-layer multi-neuron networks. A simple variation on this analysis holds also for the biases. If this is not obvious to you, then you should work through that analysis as well.

  证明： (BP1)¹ (BP2)² (BP3)³ (BP4)⁴

  若损失函数为 $C=\frac{1}{2n}{\sum }_x{\sum }_j(a_j^L-y_j{)}^2$

  由(BP1)

  ${\delta }_j^L=\frac{\partial C}{\partial a_j}{\sigma }^{\prime }(z_j)=\frac{1}{n}{\sum }_x(a_j^L-y_j){\sigma }^{\prime }(z_j^L)$

  由(BP4)

  $\frac{\partial C}{\partial w_{jk}^L}=[\frac{1}{n}{\sum }_x(a_j^L-$