残差网讲解

背景

残差网被提出：

He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2016: 770-778.

在另一篇文章中进行了理论分析：

He K, Zhang X, Ren S, et al. Identity mappings in deep residual networks[C]//European conference on computer vision. Springer, Cham, 2016: 630-645.

残差

残差在数理统计中的表示：实际观察值与估计拟合值之间的差。
$$\hat{\epsilon }=y_1-{\hat{y}}_1$$

深层网络的退化问题

1. 网络层数加深，收敛时出现退化问题。
   

输入：$x_1$
输出：$\hat{y}$
激活函数：$f(\cdot )$
计算方式：$x_i=f(x_{i-1}{w}_{i-1})$ $\hat{y}=f(x_4{w}_4)$
损失函数：$E=\frac{1}{2}(\hat{y}-y{)}^2$
求导：
$${\hat{w}}^{\ast }=\underset{\hat{w}}{\mathrm{argmin}}\frac{1}{2}(\hat{y}-y{)}^2$$
目标最优化损失函数后，求${\hat{w}}^{\ast }$
$$\begin{array}{l}\frac{\partial \hat{y}}{\partial w_1}=\frac{\partial \hat{y}}{\partial x_4}\frac{\partial x_4}{\partial x_3}\frac{\partial x_3}{\partial x_2}\frac{\partial x_2}{\partial w_1}\\ =w_4{f}^{\prime }(x_4)w_3{f}^{\prime }(x_3)w_2{f}^{\prime }(x_2)w_1{f}^{\prime }(x_1)\end{array}$$
上述为链式法则求导。

残差单元

深度残差网有许多堆叠的残差单元组成（Residual Units）, 每个单元可以被表示为下图的形式。

${\mathrm{y}}_l=h\left({\mathrm{x}}_l\right)+\mathcal{F}\left({\mathrm{x}}_l,{\mathcal{W}}_l\right)$
${\mathrm{x}}_{l+1}=f\left({\mathrm{y}}_l\right)$
${\mathrm{x}}_{l+1}={\mathrm{x}}_l+\mathcal{F}\left({\mathrm{x}}_l,{\mathcal{W}}_l\right)$
$\begin{array}{l}{\mathrm{x}}_{l+2}={\mathrm{x}}_{l+1}+\mathcal{F}\left({\mathrm{x}}_{l+1},{\mathcal{W}}_{l+1}\right)\\ ={\mathrm{x}}_l+\mathcal{F}\left({\mathrm{x}}_l,{\mathcal{W}}_l\right)+\mathcal{F}\left({\mathrm{x}}_{l+1},{\mathcal{W}}_{l+1}\right)\end{array}$

${\mathrm{x}}_L={\mathrm{x}}_l+\sum _{i=l}^{L-1}\mathcal{F}\left({\mathrm{x}}_{\mathrm{i}},{\mathcal{W}}_{\mathrm{i}}\right)$

${\mathrm{x}}_l$ 第 $l$单元的输入特征
${\mathrm{x}}_{l+1}$ 第 $l$单元的输出特征
$\mathcal{F}\left(\cdot \right)$ 残差函数
$h\left({\mathrm{x}}_l\right)={\mathrm{x}}_l$ 恒等映射
$f\left(\cdot \right)$ 一个激活函数
$L$ 表示Residual Units的数量

残差的原理

将损失表示为$\epsilon$
$\begin{array}{l}\frac{\partial \epsilon }{\partial x_l}=\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\frac{\partial {\mathrm{x}}_L}{\partial x_l}=\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\left(1+\frac{\partial {\mathrm{x}}_L}{\partial x_l}\sum _{i=l}^{L-1}\mathcal{F}\left({\mathrm{x}}_{\mathrm{i}},{\mathcal{W}}_{\mathrm{i}}\right)\right)\\ =\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}+\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\left(\frac{\partial {\mathrm{x}}_L}{\partial x_l}\sum _{i=l}^{L-1}\mathcal{F}\left({\mathrm{x}}_{\mathrm{i}},{\mathcal{W}}_{\mathrm{i}}\right)\right)\end{array}$

$\begin{array}{l}\frac{\partial \epsilon }{\partial w_1}=\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\frac{\partial {\mathrm{x}}_L}{\partial x_l}\frac{\partial x_l}{\partial w_1}=\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\left(1+\frac{\partial {\mathrm{x}}_L}{\partial x_l}\sum _{i=l}^{L-1}\mathcal{F}\left({\mathrm{x}}_{\mathrm{i}},{\mathcal{W}}_{\mathrm{i}}\right)\right)\frac{\partial x_l}{\partial w_1}\\ =\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\frac{\partial x_l}{\partial w_1}+\frac{\partial \epsilon }{\partial {\mathrm{x}}_L}\left(\frac{\partial {\mathrm{x}}_L}{\partial x_l}\sum _{i=l}^{L-1}\mathcal{F}\left({\mathrm{x}}_{\mathrm{i}},{\mathcal{W}}_{\mathrm{i}}\right)\right)\frac{\partial x_l}{\partial w_1}\end{array}$

残差相关网络结构

1. conv shortcut在浅层训练会比较好
2. dropout相当于$\lambda$
   
   上述结构对应不同的实验性能，具体应该是没有任何理论依据的。如果想更多的讨论理论依据，应该从数据分布和网络优化层面考虑。