卷积层

图像处理里的卷积实际上是数学上的互相关,而数学上的卷积是是卷积核旋转180后的互相关

数学上的互相关,即深度学习里的卷积:

数学上的卷积

前向传播

设$a^{l-1}$为$l-1$层输出,$w^l$为$l$层权重,这里符号$*$,代表深度学习里的卷积,数学上的互相关

$$z^l=a^{l-1}*W^l$$

$$\left[ \begin{matrix} a_{11}^{l-1}& a_{12}^{l-1} & a_{13}^{l-1} \\ a_{21^{l-1}} & a_{22}^{l-1} & a_{23}^{l-1} \\ a_{31}^{l-1} & a_{32}^{l-1} & a_{33}^{l-1} \end{matrix} \right] * \left[ \begin{matrix} w_{11}^l& w_{12}^l \\ w_{21}^l & w_{22}^l \\ \end{matrix} \right] =\left[ \begin{matrix} z_{11}^l& z_{12} ^l \\ z_{21}^l & z_{22}^l \\ \end{matrix} \right]$$

下面为书写简便不再标注层数,默认a为$l-1层$,w为l层
那么按$stride=1$,有:

$$z_{i,j}=\big(\sum_{m=0}^{2}\sum_{n=0}^2a_{(i,j)}\cdot w_{(i+m,j+n)}\big)+b$$
即:
$$z_{11}=a_{11}w_{11}+a_{12}w_{12}+a_{21}w_{21}+a_{22}w_{22} +b\\ z_{12}=a_{12}w_{11}+a_{13}w_{12}+a_{22}w_{21}+a_{23}w_{22}+b\\ z_{21}=a_{21}w_{11}+a_{22}w_{12}+a_{31}w_{21}+a_{32}w_{22}+b\\ z_{22}=a_{22}w_{11}+a_{23}w_{12}+a_{32}w_{21}+a_{33}w_{22}+b\\ $$

反向传播

设本层敏感度图:

$$\delta= \left[ \begin{matrix} \delta_{11}& \delta_{12} \\ \delta_{21} & \delta_{22} \\ \end{matrix} \right]$$

那么上一层敏感度图:

$$\delta^{l-1}=\frac{\partial C}{\partial z^{l-1}}=\frac {\partial C}{\partial a^{l-1}}\frac{\partial a^{l-1}}{\partial z^{l-1}}$$

而

$$\nabla a_{i,j}=\frac {\partial C}{\partial a_{(i,j)}^{l-1}}=\sum_{m,n}^{m=2,n=2}\frac{\partial C}{\partial z_{(m,n)}^l}\frac{\partial z_{(m,n)}^l}{a_{(i,j)}^{l-1}}=\sum_{m,n}^{m=2,n=2}\delta_{(m,n)}^l\frac{\partial z_{(m,n)}^l}{a_{(i,j)}^{l-1}}$$
即:
$$\begin{aligned} &\nabla a_{11}=\delta_{11}w_{11}\\ &\nabla a_{12}=\delta_{11}w_{12}+\delta_{12}w_{12}\\ &\nabla a_{13}=\delta_{12}w_{12}\\ &\nabla a_{21}=\delta_{11}w_{21}+\delta_{21}w_{11}\\ &\nabla a_{22}=\delta_{11}w_{22}+\delta_{12}w_{21}+\delta_{21}w_{12}+\delta_{22}w_{11}\\ &\nabla a_{23}=\delta_{12}w_{22}+\delta_{22}w_{12}\\ &\nabla a_{31}=\delta_{21}w_{21}\\ &\nabla a_{32}=\delta_{21}w_{22}+\delta_{22}w_{21}\\ &\nabla a_{33}=\delta_{22}w_{22} \end{aligned}$$
这里实际上可以,把第l层的敏感度图周围填充一圈0，再将卷积核翻转 $180^o$，对两者进行互相关操作,便得到 $\nabla a$,如下图所示：
$$\nabla a= \left[ \begin{matrix} \nabla a_{11}& \nabla a_{12}& \nabla a_{13} \\ \nabla a_{21} & \nabla a_{22}& \nabla a_{23} \\ \nabla a_{31} & \nabla a_{32}& \nabla a_{33} \\ \end{matrix} \right] = \left[ \begin{matrix} 0& 0 &0 &0\\ 0& \delta_{11}& \delta_{12} &0 \\ 0& \delta_{21} & \delta_{22} &0 \\ 0& 0 &0 &0\\ \end{matrix} \right] * \left[ \begin{matrix} w_{22}& w_{21} \\ w_{12} & w_{11} \\ \end{matrix} \right] =\delta^l*rot180(w^l)$$

所以上一层敏感度图:

$$\delta^{l-1}=\frac{\partial C}{\partial z^{l-1}}=\nabla a \frac{\partial a^{l-1}}{\partial z^{l-1}}=\delta^l*rot180(w^l)\bigodot \sigma(z^{l-1})$$

求权重W的梯度

$$\frac {\partial C}{\partial w_{i,j}^l}=\sum_{m,n}^{m=2,n=2}\big(\frac{\partial C}{\partial z_{m,n}^l}\frac{\partial z_{m,n}^l}{w_{i,j}^l}\big)$$
即:
$$\nabla w_{11}=\delta_{11}a_{11}+\delta_{12}a_{12}+\delta_{21}a_{21}+\delta_{22}a_{22}\\ \nabla w_{12}=\delta_{11}a_{12}+\delta_{12}a_{13}+\delta_{21}a_{22}+\delta_{22}a_{23}\\ \nabla w_{21}=\delta_{11}a_{21}+\delta_{12}a_{22}+\delta_{21}a_{31}+\delta_{22}a_{32}\\ \nabla w_{22}=\delta_{11}a_{22}+\delta_{12}a_{23}+\delta_{21}a_{32}+\delta_{22}a_{33}\\ $$
等价于:
$$\nabla w= \left[ \begin{matrix} a_{11}& a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ a_{31} & a_{32} & a_{33} \end{matrix} \right] * \left[ \begin{matrix} \delta_{11}& \delta_{12} \\ \delta_{21} & \delta_{22} \\ \end{matrix} \right] =a^{l-1}*\delta^l$$

求偏差b的梯度

$$\frac {\partial C}{\partial b^l}=\sum_{m,n}^{m=2,n=2}\big(\frac{\partial C}{\partial z_{m,n}^l}\frac{\partial z_{m,n}^l}{b^l}\big)=\sum_{m,n}^{m=2,n=2}\frac{\partial C}{\partial z_{m,n}^l}=\sum_{m,n}^{m=2,n=2}\delta_{m,n}^l$$

主要参考:
http://www.cnblogs.com/pinard/p/6494810.html
https://www.zybuluo.com/hanbingtao/note/485480