机器学习周报（RNN的梯度消失和LSTM缓解梯度消失公式推导）

摘要

在深度学习领域，循环神经网络（Recurrent Neural Network, RNN）被广泛应用于处理序列数据，特别是在自然语言处理、时间序列预测等任务中。然而，传统的RNN在长序列数据学习过程中容易出现梯度消失和梯度爆炸问题，使得模型难以捕捉长时间依赖性。梯度消失问题源于RNN的反向传播算法中，多次矩阵相乘导致梯度指数级衰减，从而影响模型性能。为解决这一问题，长短期记忆网络（Long Short-Term Memory, LSTM）应运而生。LSTM通过设计特殊的门结构（输入门、遗忘门和输出门）以及引入细胞状态的传播，有效缓解了梯度消失现象。本文推导了RNN梯度消失的数学公式，并详细说明了LSTM如何利用门结构保持梯度稳定性，从而捕捉长时间依赖。

Abstract

Recurrent Neural Networks (RNNs) are widely used in deep learning for handling sequential data, particularly in tasks such as natural language processing and time series forecasting. However, traditional RNNs often encounter the vanishing and exploding gradient problem when learning from long sequences, which hinders their ability to capture long-term dependencies. The vanishing gradient problem arises in RNNs due to multiple matrix multiplications during backpropagation, causing exponential decay of gradients and impacting model performance. To address this issue, Long Short-Term Memory (LSTM) networks were developed. LSTM alleviates gradient vanishing by introducing specially designed gate structures—input gate, forget gate, and output gate—along with a cell state that propagates through time. This paper derives the mathematical basis for the vanishing gradient in RNNs and explains how LSTM leverages gate structures to maintain gradient stability, enabling the model to capture long-term dependencies effectively.

1 RNN的梯度消失问题

• RNN的缺点

当序列太长时，容易产生梯度消失，参数更新只能捕捉到局部以来关系，没法再捕捉序列之间长期的关联或依赖关系。

如图为RNN连接，输入x，输出o（简单线性输出），权重w，s为生成状态。

根据前向传播可得：

假设使用平方误差作为损失函数，对单个时间点进行求梯度，假设再t=3时刻，损失函数为$L_3=\frac{1}{2}(Y_3-O_3{)}^2$,然后根据网络参数Wx，Ws，Wo，b1，b2等求梯度。

• $W_o$求梯度得：

• $W_x$求梯度得：

具体求解过程：首先，所求目标为$L_3$对$W_x$的偏导，通过链式法则进行展开。对比前向传播公式图可知， O 3 O_3