典型常见的知识蒸馏方法总结二

原创

已于 2024-12-07 01:18:38 修改 · 1k 阅读

27 ·

CC 4.0 BY-SA版权

文章标签：

#知识蒸馏

于 2024-12-07 01:11:20 首次发布

来源：https://github.com/open-mmlab/MMRazor收录的方法

ICCV 2021：Channel-Wise Knowledge Distillation for Dense Prediction

在这里插入图片描述

首先，使用1x1卷积将教师模型与学生模型的中间层特征的通道维度进行对齐。

A 1x1 convolution layer is employed to upsample the number of channels for the student network if the number of channels mismatches between the teacher and the student.

然后，对每个通道的WxH的特征进行softmax归一化。

在这里插入图片描述

最后，使用KL散度损失来最小化学生特征与教师特征之间的距离

在这里插入图片描述

ICLR 2018 Workshop：Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks

Adversarial learning for knowledge distillation can be first found in (Ref. ICLR 2018 Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks).
Liu et al. (Ref. CVPR 2019：Structured Knowledge Distillation for Semantic Segmentation) shares the similar idea for semantic segmentation, named holistic distillation. We also leverage the adversarial learning performed in the output space.