[CVPR 2019] Semantic Projection Network for Zero- and Few-Label Semantic Segmentation

最新推荐文章于 2024-03-21 11:28:57 发布

一亩高粱

最新推荐文章于 2024-03-21 11:28:57 发布

阅读量1.2k

点赞数

分类专栏：零样本学习

本文链接：https://blog.youkuaiyun.com/cp_oldy/article/details/100176011

版权

零样本学习专栏收录该内容

20 篇文章

订阅专栏

Zero- and Few-Label Semantic Segmentation

在这里插入图片描述 Figure 1: We propose (generalized) zero- and few-label semantic segmentation tasks, i.e. segmenting classes whose labels are not seen by the model during training or the model has a few labeled samples of those classes. To tackle these tasks, we propose a model that transfers knowledge from seen classes to unseen classes using side information, e.g. semantic word embedding trained on free text corpus.

思路

在分割网络中嵌入类别语义信息，使用辅助信息（例如基于文本语料库训练得到的语义词嵌入）将已见类的知识迁移到未见类别。

在这里插入图片描述

Figure 2: Our zero-label and few-label semantic segmentation model, i.e. SPNet, consists of two steps: visual semantic embedding and semantic projection. Zero-label semantic segmentation is drawn as an instance of our model. Replacing different components of SPNet, four tasks are addressed (Solid/dashed lines show the training/test procedures respectively).

两步：
一、视觉语义映射；
二、语义映射

域漂移校正
The extreme case of the imbalanced data problem occurs when there is no labeled training images of unseen classes, and this results in predictions being biased to seen classes. To fix this issue, we follow [8] and calibrate the prediction by reducing the scores of seen classes, which leads to:

$\arg\max_{u ∈ \mathcal {S∪U}} p({\hat y}_{ij} = u | x; [W_s ; W_u]) − γI[u ∈ \mathcal{S}] \tag 5$

where $I = 1$ if $u$ is a seen class and 0 otherwise, $γ \in [0, 1]$ is the calibration factor tuned on a held-out validation set.

在分类任务中，一整图片对应一个类别，语义信息有对应的视觉区域。那么，在分割任务中，每个类别的所有像素似乎无差异，带有语义信息的视觉区域不明显。

实验

词向量的效果

在这里插入图片描述

网络结构的效果

在这里插入图片描述

对象大小的效果

在这里插入图片描述
Figure 3: mIoU of unseen classes on COCO-Stuff ordered wrt average object size (left to right).

GZSL结果

在这里插入图片描述
Figure 4: GZLSS results on COCO-Stuff and PASCALVOC. We report mean IoU of unseen classes, seen classes and their harmonic mean (perception model is based on ResNet101 and the semantic embedding is ft + w2v). SPNet-C represents SPNet with calibration.