[Paper note] Language Modeling with Gated Convolutional Networks

最新推荐文章于 2025-09-28 11:37:32 发布

原创最新推荐文章于 2025-09-28 11:37:32 发布 · 1k 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#CNN #语言模型 #深度学习

paper-note 专栏收录该内容

20 篇文章

订阅专栏

本文介绍了一种使用CNN构建的语言模型，并通过实验证明其在多项指标上优于传统的LSTM模型。实验中对比了两种模型在Google Billion Word及WikiText-103数据集上的表现，结果显示CNN模型不仅收敛速度快，而且参数量更少，运行效率更高。

paper
CNN beats LSTM in language model?

Model

Model
* Word embedding
* Hidden layers: $h_l(X)=(X*W+b) \bigotimes \sigma(X*V+c)$
* $\bigotimes$ : element-wise product
* $W,V \in \mathbb{R}^{k\times m\times n}$
* Note that the begin of the sequence is zero-padded by k/2
* The linear gate can alleviate vanishing gradient problem (can be seen as a multiplicative skip connection)
* Adaptive softmax

Experiment

Datasets: Google Billion Word (GBW), WikiText-103
Optimization: Nesterov’s momentum, gradient clipping (to 0.1), weight normalization
- Can gain stable and fast convergence with large learning rate such as 1
Hyper-parameter search
Result
- Speed: GCNN-22 compared with LSTM-2048 (units), better throughput and responsiveness
- Complexity: GCNN-22 compared with LSTM-2048 (units), less parameter and FLOPs/token
- Gating mechanism: GLU > (GTU (LSTM unit) <=> ReLU) > Tanh
- Non-linear modeling: GLU > Linear > Bilinear
- Network depth: the deep the better
- Context: large context size brings lower test perplexity but returns diminish.