量化网络训练--Towards Effective Low-bitwidth Convolutional Neural Networks

Towards Effective Low-bitwidth Convolutional Neural Networks
CVPR2018
https://github.com/nowgood/QuantizeCNNModel

本文针对 低 bitwidth CNN网络 提出三个训练技巧 以得到较高精度。这些技巧可以独立使用也可以结合使用。

第一个技巧：首先量化 weight，得到足够好的效果后再 量化 activation
第二个技巧：逐步降低网络的位数，32-bit→8-bit→4-bit→2-bit
第三个技巧：同时训练高精度网络和低精度网络，两者相互学习 train a full-precision network alongside the target low-precision network， Guided training with a full-precision network

It is observed that by using the guidance of the teacher model, better performance can be obtained with the student model than directly training the student model on the target problem.