【论文阅读】FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning

论文下载
GitHub
bib:

@INPROCEEDINGS{
   
   wang2023freematch,
	 title		= {
   
   FreeMatch: Self-adaptive Thresholding for Semi-supervised Learning},
     author		= {
   
   Wang, Yidong and Chen, Hao and Heng, Qiang and Hou, Wenxin and Fan, Yue and and Wu, Zhen and Wang, Jindong and Savvides, Marios and Shinozaki, Takahiro and Raj, Bhiksha and Schiele, Bernt and Xie, Xing},
  	 booktitle	= {
   
   ICLR},
     year		= {
   
   2023},
     pages      = {
   
   1--20}
}

1. 摘要

1. Semi-supervised Learning (SSL) has witnessed great success owing to the impressive performances brought by various methods based on pseudo labeling and consistency regularization.

半监督学习（semi-supervised learning）的两大杀器，pseudo labeling（伪标记）和consistency regularization（一致性正则）。

2. However, we argue that existing methods might fail to utilize the unlabeled data more effectively since they either use a pre-defined / fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence.

提出现有方法的不足，use a pre-defined / fixed threshold or an ad-hoc threshold adjusting scheme。这里的阈值应该是伪标签加入的阈值，只有大于阈值的伪标签才会加入训练，这一点在基于伪标签技术的方法中十分的常用。至于基于一致性正则中是否也存在这样的阈值，这一点是存疑的。

4. We first analyze a motivating example to obtain intuitions on the relationship between the desirable threshold and model’s learning status. Based on the analysis, we hence propose FreeMatch to adjust the confidence threshold in a self-adaptive manner according to the model’s learning status.

顺利提出自己的核心创新点： self-adaptive confidence threshold。

5. We further introduce a self-adaptive class fairness regularization penalty to encourage the model for diverse predictions during the early training stage.

一个trick，避免模型初期过早收敛。

6. Extensive experiments indicate the superiority of FreeMatch especially when the labeled data are extremely rare. FreeMatch achieves 5.78%, 13.59%, and 1.28% error rate reduction over the latest state-of-the-art method FlexMatch on CIFAR-10 with 1 label per class, STL-10 with 4 labels per class, and ImageNet with 100 labels per class, respectively. Moreover, FreeMatch can also boost the performance of imbalanced SSL.

自信的算法用三句话来描述自己是art-of-state的。

7. The codes can be found at https: //github.com/microsoft/Semi-supervised-learning.

代码地址。

2. 算法描述

2.1. 例子

通过一个分类的例子，有以下有趣的结论：

1. 简单地说，未标记数据利用率（采样率）$1-P(Y_p=0)$ 直接由阈值 $\tau$ 控制。随着置信度阈值 $\tau$ 变大，未标记数据利用率变低。在训练初期，由于 $\beta$ 仍然很小，采用较高的阈值可能会导致采样率较低且收敛速度较慢。
2. 更有趣的是，如果 ${\sigma }_1\ne {\sigma }_2$，则 $P(Y_p=1)\ne P(Y_p=-1)$。事实上，$\tau$ 越大，伪标签越不平衡。从我们旨在解决平衡分类问题的意义上来说，这可能是不可取的。不平衡的伪标签可能会扭曲决策边界并导致所谓的伪标签偏差。对此的一个简单的补救措施是使用特定于类的阈值 ${\tau }_2$ 和 $1-{\tau }_1$ 来分配伪标签。(different classes have different levels of intra-class diversity (different σ))
3. 采样率 $1-P(Y_p=0)$ 随着 ${\mu }_2-{\mu }_1$ 变小而降低。换句话说，两个类越相似，未标记的样本就越有可能被屏蔽。随着两个类别变得更加相似，特征空间中会混合更多的样本，而模型对其预测的信心较差，因此需要一个适度的阈值来平衡采样率。否则，我们可能没有足够的样本来训练模型来对已经很难分类的类进行分类。(some classes are harder to classify than others (µ2 − µ1 being small)

Since different classes have different levels of intra-class diversity (different σ) and some classes are harder to classify than others (µ2 − µ1 being small), a fine-grained class-specific threshold is desirable to encourage fair assignment of pseudo labels to different classes.

2.2. Self-adaptive Threshold

Global Threshold:
$${\tau }_t=\{\begin{array}{ll}\frac{1}{C},& \text{if }t=0,\\ \lambda {\tau }_{t-1}+(1-\lambda )\frac{1}{\mu B}{\sum }_{b=1}^{\mu B}\max q_b,& \text{otherwise}.\end{array}$$

Local Threshold:
p ~ t ( c ) = { 1 C , if  t = 0 , λ p ~ t − 1 ( c ) + ( 1 − λ ) 1 μ B ∑ b = 1 μ B q b ( c ) , otherwise . \widetilde{p}_t(c)= \begin{cases} \frac{1}{C},& \text{if } t=0,\\ \lambda\widetilde{p}_{t-1}(c) + (1-\lambda)\frac{1}{\mu B}\sum_{b=1}^{\mu B}q_b(c),& \text{otherwise}. \end{cases}