Backpropagation

Backpropagation

@[深度学习, 向后传播算法]

$w_jk^l$表示$(l-1)^{th}$层的第$k$个神经元和第$l^{th}$层的第j个元素的连接

$b_j^l$第$l$层神经元的bias
$a_j^i$第$l$层神经元的activation

$$\begin{eqnarray} a^{l}_j = \sigma\left( \sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right), \tag{1}\end{eqnarray}$$

$w_{jk}^l$:$j$ 的范围是第l层神经元的个数，$k$的范围是第$(l-1)$层神经元的个数，这个表示方便将公式表示为矩阵的形式

$$\begin{eqnarray} a^{l} = \sigma(w^l a^{l-1}+b^l). \tag{2}\end{eqnarray}$$
这里$a^{l-1}$是第$l$层的激活神经元

代价函数相关的两个假设

backpropagation的目标就是计算代价函数对$w$和$b$的偏导

二次代价函数的形式：

$$\begin{eqnarray} C = \frac{1}{2n} \sum_x \|y(x)-a^L(x)\|^2, \tag{3}\end{eqnarray}$$

• 假设1：代价函数能够被写成$C = \frac {1}{n}\sum_x C_x$,需要这个假设的原因是backpropagation实际上需要我们计算的是对单个训练样本的偏导（$\frac {\partial C_x} {\partial w}$ 和 $\frac {\partial C_x} {\partial b}$）
• 假设2：代价函数能够被写成神经网络输出的函数
  
  例如：二次代价函数能够写成：

$$\begin{eqnarray} C = \frac{1}{2} \|y-a^L\|^2 = \frac{1}{2} \sum_j (y_j-a^L_j)^2, \tag{4}\end{eqnarray}$$

Hadamard product

$$\begin{eqnarray} \left[\begin{array}{c} 1 \\ 2 \end{array}\right] \odot \left[\begin{array}{c} 3 \\ 4\end{array} \right] = \left[ \begin{array}{c} 1 * 3 \\ 2 * 4 \end{array} \right] = \left[ \begin{array}{c} 3 \\ 8 \end{array} \right]. \tag{5}\end{eqnarray}$$

Backpropagation的四个基本等式

backpropagation是为了理解在神经网络中改变weights和biases是怎样改变代价函数，最终，意味着计算偏导$\frac {\partial C} {\partial w_{jk}^l}$ 和 $\frac {\partial C} {\partial b_j^l}$

为了计算偏导，我们首先计算中间量，$\delta_j^l$,表示$l^{th}$层第$j$个神经元的error

$$\begin{eqnarray} z^{l}_j = \left (\sum_k w^{l}_{jk} a^{l-1}_k + b^l_j \right) , \tag{6}\end{eqnarray}$$

$$\begin{eqnarray} \delta^l_j \equiv \frac{\partial C}{\partial z^l_j}. \tag{7}\end{eqnarray}$$

backpropagation给了一个对每层计算$\delta^l$的方法

误差在输出层的等式

$$\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j). \tag{BP1}\end{eqnarray}$$
* $\partial C / {\partial a_j^L}$衡量了以第$j^{th}$个激活元为函数的变化速率
* $\sigma'(z^L_j)$衡量了sigmoid函数对$z_j^L$的变化速率
* $\partial C / {\partial a_j^L}$的精确形式取决于代价函数的选择，例如针对二次代价函数而言
* 当$\sigma(z^L_j)$趋近于0或1的时候，$\sigma'(z^L_j) \approx 0$，$\delta^L_j$也会变的很小，可以说输出神经元已经饱和，weight开始停止学习或学习的很慢

$C= \frac {1} {2} \sum_j (y_j - a_j)^2$
$\frac {\partial C }{ \partial a_j^L} = (a_j - y_j)$

$$\begin{eqnarray} \delta^L = \nabla_a C \odot \sigma'(z^L). \tag{BP1a}\end{eqnarray}$$
二次代价函数的 $\delta^L$
$$\delta^L = (a^L - y) \odot \sigma'(z^L)$$

在式子中都有较好的向量形式，因此容易利用Numpy等库进行计算

下一层误差等式,$\delta^{l+1}$

$$\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l), \tag{BP2}\end{eqnarray}$$

总结而言：
* 当输出神经元的状态是low-activation或者sturated时，weight将会缓慢的学习
* 这四个公式对任何形式的激活函数都有用

An equation for the rate of change of the cost with respect to any bias in the network:

$$\frac {\partial C} {\partial b^l_j} = \delta ^l _j\tag {BP3}$$

$$\begin{eqnarray} \frac{\partial C}{\partial b} = \delta, \tag{BP3a}\end{eqnarray}$$
An equation for the rate of change of the cost with respect to any weight in the network
$$\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j. \tag{BP4}\end{eqnarray}$$

$$\begin{eqnarray} \frac{\partial C}{\partial w} = a_{\rm in} \delta_{\rm out}, \tag{BP4a}\end{eqnarray}$$

证明

（BP1）

$$\begin{eqnarray} \delta^L_j = \frac{\partial C}{\partial a^L_j} \sigma'(z^L_j). \tag{BP1}\end{eqnarray}$$

$$\delta^L_j = \frac{\partial C}{\partial z^L_j}$$
链式法则
$$\delta^L_j = \frac {\partial C}{\partial a^L_j} \frac {\partial a^L_j} {\partial z^L_j} \\$$

(BP2)

$$\begin{eqnarray} \delta^l = ((w^{l+1})^T \delta^{l+1}) \odot \sigma'(z^l), \tag{BP2}\end{eqnarray}$$

$$\begin{align} \delta^l_j &= \frac{\partial C}{\partial z^l_j} \tag{链式法则} \\ &= \sum_k \frac{\partial C}{\partial z^{l+1}_k} \frac{\partial z^{l+1}_k}{\partial z^l_j} \tag{a}\\ &= \sum_k \frac{\partial z^{l+1}_k}{\partial z^l_j} \delta^{l+1}_k, \tag{b}\end{align}$$
$$z^{l+1}_k = \sum_j w^{l+1}_{kj} a^l_j +b^{l+1}_k = \sum_j w^{l+1}_{kj} \sigma(z^l_j) +b^{l+1}_k$$

(BP3)

$$\frac {\partial C} {\partial b^l_j} = \delta ^l _j\tag {BP3}$$

$$\begin{align}\frac {\partial C} {\partial b^l_j} = \frac {\partial C} {\partial z^l_j} \frac {\partial z^l_j}{\partial b^l_j} \end{align}$$
$$\frac {\partial z^l_j}{\partial b^l_j} = 1$$

(BP4)

$$\begin{eqnarray} \frac{\partial C}{\partial w^l_{jk}} = a^{l-1}_k \delta^l_j. \tag{BP4}\end{eqnarray}$$

$$\begin{align}\frac {\partial C} {\partial w^l_{jk}} = \frac {\partial C} {\partial z^l_j} \frac {\partial z^l_j}{\partial w^l_{jk}} \tag{链式法则}\end{align}$$
$$z^l_j = \sum_k {w^l_{jk} a^{l-1}_k} +b_j^l$$

backpropagation算法

backpropagation equations 提供了一个计算代价函数梯度的方式

1. 输入 x :设置相应的 activation $a^1$为输入层
2. Feedforward： 对每一层$l = 2,3,...,L$计算$z^l = w_la^{l-1} + b^l$ 和 $a^l = \sigma (z^l)$
3. Output error (输出层误差) ： 计算向量$\delta^L = \nabla_a C \odot \sigma'(z^L)$
4. Backpropagate the error : 对每一层$l = L-1,L-2,...,2$计算$\delta^l = ((w^{l+1})^T\delta^{l+1}) \odot \sigma'(z^l)$
5. 输出 : 计算代价函数的梯度，通过 $\frac {\partial C}{\partial w^l_{jk}} = a_k^{l-1} \delta^l_j 和 \frac {\partial C}{\partial b^l_j} = \delta_l^j$

mini-batch:(随机梯度下降结合backpropagation)
1. 输入一组训练样本
2. 对每个训练样本：设置相应的输入激活元$a^{x,1}$
- Feedforward： 对每一层$l = 2,3,...,L$计算$z^{x,l} = w_la^{x,l-1} + b^l$ 和 $a^{x,l} = \sigma (z^{x,l})$
- Output error (输出层误差) ： 计算向量$\delta^{x,L} = \nabla_a C_x \odot \sigma'(z^{x,L})$
- Backpropagate the error : 对每一层$l = L-1,L-2,...,2$计算$\delta^{x,l} = ((w^{l+1})^T\delta^{x,l+1}) \odot \sigma'(z^{x,l})$
3. 梯度下降 ： 对每一层 $l = L,L-1,...,2$,更新权重weights，根据规则

$$w^l \rightarrow w^l - \frac {\eta}{m} \sum_x \delta^{x,l}(a^{x,l-1})^T$$ $$b^l \rightarrow b^l - \frac {\eta}{m} \sum_x \delta^{x,l}$$