WS-DAN:Weakly Supervised Data Augmentation Netowrk for Fine-Grained Visual Classification

最新推荐文章于 2024-09-15 00:31:34 发布

原创最新推荐文章于 2024-09-15 00:31:34 发布 · 802 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#深度学习 #数据扩充 #细粒度分类

Paper Notes 专栏收录该内容

11 篇文章

订阅专栏

本文提出一种弱监督数据增强网络(WS-DAN)，通过生成注意力图表示对象的判别部分，并指导数据增强过程，包括注意力裁剪和注意力丢弃，以提高细粒度视觉分类的效率和准确性。

See Better Before Looking Closer: Weakly Supervised Data Augmentation Netowrk for Fine-Grained Visual Classification
Paper PDF

Abstract

In practice, random data augmentation, such as random image cropping, is low-efficiency and might introduce many uncontrolled background noises. In this paper, they propose Weakly Supervised Data Augmentation Network (WS-DAN) to explore the potential of data augmentation. Specifically, for each training image, we first generate attention maps to represent the object’s discriminative parts by weakly supervised learning. Next, we augment the image guided by these attention maps, including attention cropping and attention dropping. The proposed WS-DAN improves the classification accuracy in two folds. In the first stage, images can be seen better since more discriminative parts’ features will be extracted. In the second stage, attention regions provide accurate location of object, which ensures our model to look at the object closer and further improve the performance.

In summary, the main contributions of this work are:

They propose Weakly Supervised Attention Learning to generate attention maps to represent the spatial distribution of discriminative object’s parts, And use BAP module to get the whole object feature by accumulating the object’s part feature.
Based on attention maps, they propose attention-guided data augmentation to improve the efficiency of data augmentation, including attention cropping and attention dropping. Attention cropping randomly crops and resizes one of the attention part to enhance the local feature representation. Attention dropping randomly erases one of the attention region out of the image in order to encourage the model to extract the feature from multiple discriminative parts.
They utilize attention maps to accurately locate the whole object and enlarge it to further improve the classification accuracy.

Innovation

Bilinear Attention Pooling(BAP)
Attention Regularization
Attention-guided Data Augmentation

Pipeline

在这里插入图片描述
The training process can be divided into two parts: Weakly Supervised Attention Learning and Attention-guided Data Augmentation:

Weakly Supervised Attention Learning

Spatial Representation

Attention maps A which is obtained from Feature maps F by convolutional function $f(\cdot)$ in Equ 1 . Each Attention map $A_{k}$ represents one of the object’s part or visual pattern, such as the head of a bird, the wheel of a car or the wing of an aircraft. Attention maps will be utilized to augment training data.
$A=f(F)=\bigcup_{k=1}^{M}A_{k} \tag{1}$

Bilinear Attention Pooling(BAP)

在这里插入图片描述
They propose Bilinear Attention Pooling (BAP) to extract features from these parts are represented by Attention maps. We element-wise multiply feature maps F by each attention map $A_{k}$ in order to generate M part feature maps $F_{k}$ , as shown in Equ 2
$F_{k} = A_{k} \odot F (k = 1, 2, ...,M) \tag{2}$

Then, They further extract discriminative local feature by additional feature extraction function $g(\cdot)$ such as Global Average Pooling (GAP), Global Maximum Pooling (GMP) or convolutions, in order to obtain $k_{th}$ attention feature $f_{k}$ .
$f_{k}=g(F_{k}) \tag{3}$

Object’s feature is represented by part feature matrix P which is stacked by these part features $f_{k}$ .

$P=\begin{pmatrix} g(a_{1} \odot F) \\ g(a_{2} \odot F) \\ ... \\ g(a_{M} \odot F) \end{pmatrix} =\begin{pmatrix} f_{1} \\ f_{2} \\ ... \\ f_{M} \end{pmatrix} \tag{4}$

Attention Regularization

For each fine-grained category, They expect that attention map $A_{k}$ can represent the same $k_{th}$ object’s part. They penalize the variances of features that belong to the same object’s part, which means that part feature $f_{k}$ will get close to the a global feature center $c_{k}$ and attention map $A_{k}$ will be activated in the same $k_{th}$ object’s part. The loss function can be represented by $L_{A}$ in Equ 5.
$L_{A}=\sum_{k=1}^{M}\left \| f_{k} - c_{k} \right \|_{2}^{2} \tag{5}$

$c_{k}$ wil updates by the Equ 6 from initialization zero.
$c_{k} \leftarrow c_{k} + \beta(f_{k} -c_{k}) \tag{6}$

Attention-guided Data Augmentation

Random image cropping, is low-efficiency and a high percentage of them contain many background noises, which might lower the training efficiency, affect the quality of the extracted features and cancel out its benefits. Using Attention as guideline, the crop images may focus more on the target.

Augmentation Map

For each training image, they randomly choose one of its attention map $A_{k}$ to guide the data augmentation process, and normalize it as $k_{th}$ Augmentation Map $A_{k}^{*}$
$A_{k}^{*} = \frac{A_{k}-min(A_{k})}{max(A_{k})-min(A_{k})} \tag{7}$

Attention Cropping

Crop Mask $C_{k}$ from $A_{k}^{*}$ by setting element $A_{k}^{*}(i,j)$ which is greater than threshold $\theta_{c}$ to 1, and others to 0, as represented in Equ 8.

$C_{k}(i,j)={\begin{cases} 1, & \text{ if } A_{k}^{*}(i,j) > \theta_{c} \\ 0, & \text {otherwise.} \end{cases}} \tag{8}$

We then find a bounding box Bk that can cover the whole
selected positive region of $C_{k}$ and enlarge this region from raw image as the augmented input data.
在这里插入图片描述

Attention Dropping

To encourage attention maps represent multiple discriminative object’s parts, they propose attention dropping. Specifically, they obtain attention Drop Mask $D_{k}$ by setting element $A_{k}^{*}(i,j)$ which is greater than threshold $\theta_{d}$ to 0, and others to 1, as shown in Equ 9

$D_{k}(i,j)={\begin{cases} 1, & \text{ if } A_{k}^{*}(i,j) > \theta_{d} \\ 0, & \text {otherwise.} \end{cases}} \tag{9}$
在这里插入图片描述

Object Localization and Refinement

In the testing process, after the model outputs the coarse-stage classification result and corresponding attention maps for the raw image, we can predict the whole region of the object and enlarge it to predict fine-grained result by the same network model. Object Map $A_{m}$ that indicates the location of object is calculated by Equ 10.
$A_{m}=\frac{1}{M}\sum_{k=1}^{M} A_{k} \tag{10}$
The final classification result is averaged by the coarse- grained prediction and fine-grained prediction. The detailed process of Coarse-to-Fine prediction is described as Algorithm below:
在这里插入图片描述