[Paper note] Language Modeling with Gated Convolutional Networks

最新推荐文章于 2024-10-13 17:55:06 发布

chn13

最新推荐文章于 2024-10-13 17:55:06 发布

阅读量1k

点赞数

CC 4.0 BY-SA版权

分类专栏： paper-note 文章标签： CNN 语言模型深度学习

本文链接：https://blog.youkuaiyun.com/chn13/article/details/54090834

paper-note 专栏收录该内容

20 篇文章

订阅专栏

本文介绍了一种使用CNN构建的语言模型，并通过实验证明其在多项指标上优于传统的LSTM模型。实验中对比了两种模型在Google Billion Word及WikiText-103数据集上的表现，结果显示CNN模型不仅收敛速度快，而且参数量更少，运行效率更高。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

paper
CNN beats LSTM in language model?

Model

Model
* Word embedding
* Hidden layers: $h_l(X)=(X*W+b) \bigotimes \sigma(X*V+c)$
* $\bigotimes$ : element-wise product
* $W,V \in \mathbb{R}^{k\times m\times n}$
* Note that the begin of the sequence is zero-padded by k/2
* The linear gate can alleviate vanishing gradient problem (can be seen as a multiplicative skip connection)
* Adaptive softmax

Experiment

Datasets: Google Billion Word (GBW), WikiText-103
Optimization: Nesterov’s momentum, gradient clipping (to 0.1), weight normalization
- Can gain stable and fast convergence with large learning rate such as 1
Hyper-parameter search
Result
- Speed: GCNN-22 compared with LSTM-2048 (units), better throughput and responsiveness
- Complexity: GCNN-22 compared with LSTM-2048 (units), less parameter and FLOPs/token
- Gating mechanism: GLU > (GTU (LSTM unit) <=> ReLU) > Tanh
- Non-linear modeling: GLU > Linear > Bilinear
- Network depth: the deep the better
- Context: large context size brings lower test perplexity but returns diminish.