【2】Model Compression As Constrained Optimization, with Application to Neural Nets. 2017

文章地址: https://arxiv.org/pdf/1707.01209.pdf

Part I: general framework.

We give a general formulation of model compression as constrained optimization.

Related work.

Four categories of model compression.

1. Direct learning: ${\min }_{\Theta }L(h(x;\Theta ))$: find the small model with the best loss regardless of the reference.
2. Direct compression (DC): ${\min }_{\Theta }\Vert w-∆(\Theta ){\Vert }^2$: find the closest approximation to the parameters of the reference model.
3. Model compression as constrained optimization: It forces $h$ and $f$ to be models of the same type, by constraining the weights $w$ to be constructed from a low-dimensional parameterization $w=∆(\Theta )$, but $h$ must optimize the loss $L$.
4. Teacher-student : ${\min }_{\Theta }{\int }_{X}p(x)\Vert f(x;w)-h(x;\Theta ){\Vert }^{2}dx$: find the closest approximation $h$ to the reference function $f$, in some norm.

A constrained optimization formulation.

Types of compression

A “Learning-Compression” (LC) algorithm

Direct compression (DC) and the beginning of the path

Compression, generalization and model selection

Compression
Compression can also be seen as a way to prevent overfitting, since it aims at obtaining a smaller model with a similar loss to that of a well-trained reference model.

Generalization
The reference model was not trained well enough, so that the continued training that happens while compressing reduces the error.

Model selection
A good approximate strategy for model selection in neural nets is to train a large enough reference model and compress it as much as possible.