Training Very Deep Networks论文笔记

Abstract
Theoretical and empirical evidence indicates that the depth of neural networks is crucial for their success. However, training becomes more difficult as depth increases, and training of very deep networks remains an open problem. Here we introduce a new architecture designed to overcome this. Our so-called highway networks allow unimpeded information flow across many layers on information highways. They are inspired by Long Short-Term Memory recurrent networks and use adaptive gating units to regulate the information flow. Even with hundreds of layers, highway networks can be trained directly through simple gradient descent. This enables the study of extremely deep and efficient architectures.

摘要
理论和实证证据表明，神经网络的深度对其性能至关重要。 然而，随着深度的增加，训练变得更加困难，这对于深度网络的训练来说仍然是一个悬而未决的问题。 在这里，我们介绍一种旨在克服这一点的新架构。 我们称其为高速公路网络，此网络允许信息在高速公路网络的多层中畅通无阻。 这个网络是受到LSTM的启发，并使用自适应门控单元来调节信息流。 即使有数百层，也可以通过简单的梯度下降直接训练高速公路网络。 这使得研究极其深入和高效的架构成为可能。

2 Highway Networks
Notation We use boldface letters for vectors and matrices, and italicized capital letters to denote transformation functions. 0 and 1 denote vectors of zeros and ones respectively, and I denotes an identity matrix. The function σ(x) is defined as $\sigma \left(x\right)=\frac{1}{1+e^{-x}}$;$xϵR$. The dot operator (·) is used to denote element-wise multiplication.
A plain feedforward neural network typically consists of L layers where the $l^{th}$ layer (lϵ{1,2,...,L}l\epsilon \left \{ 1,2,...,L \right \}lϵ{1,2,...,L}) applies a non-linear transformation H (parameterized by $W_{H,l}$) on its input $x_l$ to produce its output $y_l$. Thus, $x_1$ is the input to the network and $y_L$ is the network’s output. Omitting the layer index and biases for clarity,
$$y=H\left(x,W_H\right)$$ (1)
H is usually an affine transform followed by a non-linear activation function, but in general it may take other forms, possibly convolutional or recurrent. For a highway network, we additionally define two non-linear transforms $T\left(x,W_T\right)$and $C\left(x,W_C\right)$ such that
$$y=H\left(x,W_H\right)\cdot T\left(x,W_T\right)+x\cdot C\left(x,W_C\right)$$ (2)
We refer to T as the transform gate and C as the carry gate, since they express how much of the output is produced by transforming the input and carrying it, respectively. For simplicity, in this paper we set C = 1 − T, giving
$$y=H\left(x,W_H\right)\cdot T\left(x,W_T\right)+x\cdot (1-T\left(x,W_T\right))$$ (3)
The dimensionality of x; y; $H\left(x,W_H\right)$ and $T\left(x,W_T\right)$ must be the same for Equation 3 to be valid.
Note that this layer transformation is much more flexible than Equation 1. In particular, observe that for particular values of T,
$$y=\{\begin{array}{cc}x,& ifT\left(x,W_T\right)=0\\ H\left(x,W_H\right),& ifT\left(x,W_T\right)=1\end{array}$$ (4)

Similarly, for the Jacobian of the layer transform,
$$\frac{dy}{dx}=\{\begin{array}{cc}I& ifT\left(x,W_T\right)=0\\ H^{{}^{\prime }}\left(x,W_H\right)& ifT\left(x,W_T\right)=1\end{array}$$(5)
Thus, depending on the output of the transform gates, a highway layer can smoothly vary its behavior between that of H and that of a layer which simply passes its inputs through. Just as a plain layer consists of multiple computing units such that the $i^{th}$ unit computes $y_i=H_i(x)$, a highway network consists of multiple blocks such that the $i^{th}$ block computes a block state $H_i(x)$ and transform gate output $T_i(x)$. Finally, it produces the block output $y_i=H_i(x)\ast T_i(x)+x_i\ast (1-T_i(x))$, which is connected to the next layer.2