Exploring Sparsity in Recurrent Neural Networks

最新推荐文章于 2020-12-09 10:09:04 发布

转载最新推荐文章于 2020-12-09 10:09:04 发布 · 1.1k 阅读

DL 同时被 2 个专栏收录

158 篇文章

订阅专栏

paper reading

85 篇文章

订阅专栏

本文提出了一种在初始训练过程中通过剪枝权重来减少循环神经网络(RNN)参数的技术。训练结束后，网络参数变得稀疏，同时保持接近原始密集神经网络的准确性。这种方法可以将网络大小减少8倍，并且训练时间保持不变。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Exploring Sparsity in Recurrent Neural Networks

Sharan Narang, Gregory Diamos, Shubho Sengupta, Erich Elsen

(Submitted on 17 Apr 2017)

Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8x and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around 2x to 7x.

Comments:	Published as a conference paper at ICLR 2017
Subjects:	Learning (cs.LG); Computation and Language (cs.CL)
Cite as:	arXiv:1704.05119 [cs.LG]
	(or arXiv:1704.05119v1 [cs.LG] for this version)