RNN网络结构及公式推导

RNN结构如图所示：

$X^t\in R^x$表示t时刻的输入($X^t$是多少维，则这一层有多少个神经元，这里设为$x$维，图中画的是3维）

$h^t\in R^h$表示t时刻隐层的输出(假设这一层有h个神经元）

$y^t\in R^y$表示t时刻的预测输出
$d^t\in R^y$表示t时刻的期望输出
$V \in R^{x\times h}$表示从输入层到隐层的权值矩阵
$U\in R^{h\times h}$表示上一个时刻到这个时刻的权值矩阵
$b^h\in R^h$表示隐层的偏置，其中每一项对应某一神经元的偏置项
$W\in R^{h\times y}$表示隐层到输出层的权值矩阵
$b^y\in R^y$表示输出层的偏置项

正向传播过程：
$X^t_i$表示t时刻某个样本第i维的输入，即输入层第i个神经元的输入
t时刻隐层第j个神经元的输入：$c^t_{hj}=\sum_{i=1}^xX^t_iV_{ij}+\sum_{s=1}^hh^{t-1}_sU_{sj}+b^h_j$
t时刻隐层第j个神经元的输出:$h^t_j=f(c_{hj})$
t时刻输出层第k个神经元的输入:$c^t_{yk}=\sum_{j=1}^hh^t_jW_{jk}+b^y_k$
t时刻输出层第k个神经元的输出：$y^t_j=g(c_{yk})$

矩阵表示(只有一个样本的情况）:
t时刻隐层的输入，h*1向量，$c^t_h=V^TX^t+U^Th^{t-1}+b^h$
t时刻隐层的输出，h*1向量，$h^t=f(c^t_h)$
t时刻输出层的输入，y*1向量，$c^t_y=W^Th^t+b^y$
t时刻输出层的输出，y*1向量，$y^t=g(c^t_y)$

反向求导过程：
假设共有p个样本，则t时刻的误差可以定义为：$E^t=\sum_p\frac{1}{2}\|d^t-y^t\|^2$,整个网络的误差为$E=\sum_t E^t=\frac{1}{2}\sum_p\sum_{t=1}^T\|d^t-y^t\|^2$,

$\frac{\partial {E}}{\partial {W}}=\sum_{t=1}^T\frac{\partial {E}}{\partial {y^t}}\frac{\partial {y^t}}{\partial {W}}$

$\frac{\partial {E}}{\partial {y^t}}=-(d^t-y^t)$

$\frac{\partial {y^t}}{\partial {W}}=\frac{\partial {y^t}}{\partial {c^t_y}}\frac{\partial {c_y^t}}{\partial {W}}=g'(c_y^t)h^t$

所以$\frac{\partial {E}}{\partial {W}}=-\sum_{t=1}^T(d^t-y^t)g'(c_y^t)h^t$

$\frac{\partial {E}}{\partial {U}}=\sum_{t=1}^T\frac{\partial {E}}{\partial {h^t}}\frac{\partial {h^t}}{\partial {U}}$，$\frac{\partial {E}}{\partial {V}}=\sum_{t=1}^T\frac{\partial {E}}{\partial {h^t}}\frac{\partial {h^t}}{\partial {V}}$

由于$h^t$一方面输到$y^t$，一方面输到$h^{t+1}$,所以它的误差来自两方面：

$\frac{\partial {E}}{\partial {h^t}}=\frac{\partial {E}}{\partial {y^t}}\frac{\partial {y^t}}{\partial {h^t}}+\frac{\partial {E}}{\partial {h^{t+1}}}\frac{\partial {h^{t+1}}}{\partial {h^t}}=\frac{\partial {E}}{\partial {y^t}}\frac{\partial {y^t}}{\partial {c_y^t}}\frac{\partial {c_y^t}}{\partial {h^t}}+\frac{\partial {E}}{\partial {h^{t+1}}}\frac{\partial {h^{t+1}}}{\partial {c_h^{t+1}}}\frac{\partial {c_h^{t+1}}}{\partial {h^t}}=\frac{\partial {E}}{\partial {y^t}}g'(c_y^t)W+\frac{\partial {E}}{\partial {h^{t+1}}}f'(c_h^{t+1})U$

$\frac{\partial {h^t}}{\partial {U}}=\frac{\partial {h^t}}{\partial {c_h^t}}\frac{\partial {c_h^t}}{\partial {U}}=f'(c_h^t)h^{t-1}$，$\frac{\partial {h^t}}{\partial {V}}=\frac{\partial {h^t}}{\partial {c_h^t}}\frac{\partial {c_h^t}}{\partial {V}}=f'(c_h^t)X^t$

所以

$\frac{\partial {E}}{\partial {U}}=\sum_{t=1}^T[\frac{\partial {E}}{\partial {y^t}}g'(c_y^t)W+\frac{\partial {E}}{\partial {h^{t+1}}}f'(c_h^{t+1})U]f'(c_h^t)h^{t-1}$

$\frac{\partial {E}}{\partial {V}}=\sum_{t=1}^T[\frac{\partial {E}}{\partial {y^t}}g'(c_y^t)W+\frac{\partial {E}}{\partial {h^{t+1}}}f'(c_h^{t+1})U]f'(c_h^t)X^t$

$\frac{\partial {E}}{\partial {b^y}}=\sum_{t=1}^T\frac{\partial {E}}{\partial {y^t}}\frac{\partial {y^t}}{\partial {c_y^t}}\frac{\partial {c_y^t}}{\partial b^y}=-\sum_{t=1}^T(d^t-y^t)g'(c_y^t)$

$\frac{\partial {E}}{\partial {b^h}}=\sum_{t=1}^T\frac{\partial {E}}{\partial {h^t}}\frac{\partial {h^t}}{\partial {c_h^t}}\frac{\partial {c_h^t}}{\partial {b^h}}=\sum_{t=1}^T[\frac{\partial {E}}{\partial {y^t}}g'(c_y^t)W+\frac{\partial {E}}{\partial {h^{t+1}}}f'(c_h^{t+1})U]f'(c_h^t)$