Self-Attention Generative Adversarial Networks (SAGAN) 笔记

概述

该方法在生成器和判别器中引入self-attention。self-attention 模块是对卷积的补充，有助于建模 long range dependency。 这样，生成图像的时候就可以使用各个位置的信息，并且判别器也可以判别远距离位置之间的一致性。

The self-attention module is complementary to convolutions and helps with modeling long range, multi-level dependencies across image regions. Armed with self-attention, the generator can draw images in which fine details at every location are carefully coordinated with fine details in distant portions of the image. Moreover, the discriminator can also more accurately enforce complicated geometric constraints on the global image structure.

Self-Attention Generative Adversarial Networks

注意力的计算：

注意力层的输出：

类似残差结构，attention层的输出加上输入：

训练目标：
In SAGAN, the proposed attention module has been applied to both generator and discriminator, which are trained in an alternating fashion by minimizing the hinge version of the adversarial loss

测试生成图像质量的指标

Inception score

Fréchet Inception distance

稳定GAN训练的tricks

文章指出下面两个技巧是有效的。

Spectral normalization

这个技巧用在generator和discriminator，用于加速训练和稳定训练。

two-timescale update rule (TTUR)

用于加速regularized discriminator的训练。

文章中出现的一些名词

wall-clock time

对比wall-clock time和cpu time来理解。
Wall-clock time is the time that a clock on the wall (or a stopwatch in hand) would measure as having elapsed between the start of the process and ‘now’.
The user-cpu time and system-cpu time is the amount of time spent in user code and the amount of time spent in kernel code.
The wall-clock time is not the number of seconds that the process has spent on the CPU; it is the elapsed time, including time spent waiting for its turn on the CPU (while other processes get to run).