RNN, LSTM 图文详解

最新推荐文章于 2022-11-24 15:30:15 发布

原创

最新推荐文章于 2022-11-24 15:30:15 发布 · 1.7k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#自然语言处理

本文详细介绍了RNN和LSTM在处理序列数据时的作用。RNN因其链式结构适合序列任务，但存在梯度消失问题。LSTM通过门控机制解决了这一问题，能够学习长期依赖。此外，还提到了双向RNN和深度RNN等变体。

文章目录

往期文章链接目录

Sequence Data

There are many sequence data in applications. Here are some examples

Machine translation
- from text sequence to text sequence.
Text Summarization
- from text sequence to text sequence.
Sentiment classification
- from text sequence to categories.
Music Generation
- from nothing or some simple stuff (character, integer, etc) to wave sequence.
Name entity recognition (NER)
- From text sequence to label sequence.

Why not use a standard neural network for sequence tasks

Inputs, outputs can be different lengths in different examples. This can be solved by standard neural network by paddings with the maximum lengths but it’s not a good solution since there would be too many parameters.
Doesn’t share features learned across different positions of text/sequence. Note that Convolutional Neural Network (CNN) is a good example of parameter sharing, so we should have a similar model for sequence data.

RNN

A Recurrent Neural Network (RNN) can be thought of as multiple copies of the same network, each passing a message to a successor. This chain-like nature reveals that recurrent neural networks are intimately related to sequences. Therefore, they’re the natural architecture of neural network to use for sequence data. Note that it also allows previous outputs to be used as inputs.

For each time step $t$ , the activation $a^{<t>}$ and the output $y^{<t>}$ are expressed as follows:

$a^{<t>}=g_{1}\left(W_{a a} a^{<t-1>}+W_{a x} x^{<t>}+b_{a}\right)$
$y^{<t>}=g_{2}\left(W_{y a} a^{<t>}+b_{y}\right)$

These calculation could be visualized in the following figure

Note:

dimension of $W_{a a}$ : (number of hidden neurons, number of hidden neurons)
dimension of $W_{a x}$ : (number of hidden neurons, length of $x$ )
dimension of $W_{y a}$ : (length of $y$ , number of hidden neurons)
The weight matrix $W_{a a}$ is the memory the RNN is trying to maintain from the previous layers.
dimension of $b_a$ : (number of hidden neurons, 1)
dimension of $b_y$