（pytorch进阶之路）Attention-based U-net实现

U-net 原理部分之前的博客有些了，这里主要记录一下代码实现
U-net往期博客：https://blog.youkuaiyun.com/qq_19841133/article/details/126927383

基于Attention-based（用的是自注意力机制）的U-net
代码来源IDDPM项目：
https://github.com/openai/improved-diffusion/blob/main/improved_diffusion/unet.py

---

IDDPM的NN模型用的是attention-based Unet

Unet很熟悉了，除了有两部分编码器和解码器（input和output），还有mid block中间模块，如有ResBlock，MHSA Block

input block组成：Res（接收输入x和emb timestep表示成emb，condition表示成emb），MHSA（像素对像素的注意力机制），Downsample
mid block：Res，MHSA， Res
output block：Res（与input block对应层的输出进行拼接），MHSA，Upsample

U-net

第一个模块，time_emb，对输入进来的time_step进行变换，

		time_embed_dim = model_channels * 4
        self.time_embed = nn.Sequential(
            linear(model_channels, time_embed_dim),
            SiLU(),
            linear(time_embed_dim, time_embed_dim),
        )

如果是条件式生成那么还有一个label_emb，作为条件的embeding和x一起输入

        if self.num_classes is not None:
            self.label_emb = nn.Embedding(num_classes, time_embed_dim)

input_block，U_net的左边的那部分，把所有的Module放入ModuleList，ModuleList实例化参数一般是List，List每个元素是一个Module，第一个Module是TimestepEmbedSequential，此时List只有一个Module，后面会慢慢append进来，TimestepEmbedSequential里面是conv_nd，dims=1就是1d，dim=2就是2d，默认kernelsize是3，padding=1 （TimestepEmbedSequential见下面，其实就一个nn.Sequential的封装，其起到的作用就是选择是否传入emb，像conv_nd不用，只有ResBlock会传）

		self.input_blocks = nn.ModuleList(
            [
                TimestepEmbedSequential(
                    conv_nd(dims, in_channels, model_channels, 3, padding=1)
                )
            ]
        )

接下来对U-net左边进行搭建，首先是对channel_mult（通道乘子）进行遍历，就是乘以几倍几倍，通道乘子其实就是定义了有几层的U-net，一般是逐层扩大，输出通道数是mult * model_channels，乘子和当前model通道数的乘积，通道数正在扩大

每个乘子中有很多的Resblock，对Resblock进行遍历
ResBlock继承自TimestepEmbedSequential，那么就需要传入一个emb，这里传了time-embed-dim

 		for level, mult in enumerate(channel_mult):
            for _ in range(num_res_blocks):
                layers = [
                    ResBlock(
                        ch,
                        time_embed_dim,
                        dropout,
                        out_channels=mult * model_channels,
                        dims=dims,
                        use_checkpoint=use_checkpoint,
                        use_scale_shift_norm=use_scale_shift_norm,
                    )
                ]

我们还在num_res_blocks的遍历中，下面是创建attention-block，ds就是我们下采样的比例，如果ds在attention_resolutions的列表中，我们就插入一个AttentionBlock，所以具体在哪些地方插入取决与attention_resolutions

				ch = mult * model_channels
                if ds in attention_resolutions:
                    layers.append(
                        AttentionBlock(
                            ch, use_checkpoint=use_checkpoint, num_heads=num_heads
                        )
                    )
                self.input_blocks.append(TimestepEmbedSequential(*layers))
                input_block_chans.append(ch)

最后遍历完num_res_blocks，我们再跟一个下采样层

            if level != len(channel_mult) - 1:
                self.input_blocks.append(
                    TimestepEmbedSequential(Downsample(ch, conv_resample, dims=dims))
                )
                input_block_chans.append(ch)
                ds *= 2

U-net中间部分就是两个ResBlock和一个AttentionBlock构成
通道数和空间数都没有改变

		self.middle_block = TimestepEmbedSequential(
            ResBlock(
                ch,
                time_embed_dim,
                dropout,
                dims=dims,
                use_checkpoint=use_checkpoint,
                use_scale_shift_norm=use_scale_shift_norm,
            ),
            AttentionBlock(ch, use_checkpoint=use_checkpoint, num_heads=num_heads),
            ResBlock(
                ch,
                time_embed_dim,
                dropout,
                dims=dims,
                use_checkpoint=use_checkpoint,
                use_scale_shift_norm=use_scale_shift_norm,
            ),
        )

U-net右边，和左边镜像，由ResBlock和Attentionblock，和UpSample构成

注意的是ResBlock输入的通道数是ch + input_block_chans.pop(),因为U-net左右两边是连起来的，所以通道数应该是两者之和

		self.output_blocks = nn.ModuleList([])
        for level, mult in list(enumerate(channel_mult))[::-1]:
            for i in range(num_res_blocks + 1):
                layers = [
                    ResBlock(
                        ch + input_block_chans.pop(),
                        time_embed_dim,
                        dropout,
                        out_channels=model_channels * mult,
                        dims=dims,
                        use_checkpoint=use_checkpoint,
                        use_scale_shift_norm=use_scale_shift_norm,
                    )
                ]
                ch = model_channels * mult
                if ds in attention_resolutions:
                    layers.append(
                        AttentionBlock(
                            ch,
                            use_checkpoint=use_checkpoint,
                            num_heads=num_heads_upsample,
                        )
                    )
                if level and i == num_res_blocks:
                    layers.append(Upsample(ch, conv_resample, dims=dims))
                    ds //= 2
                self.output_blocks.append(TimestepEmbedSequential(*layers))

最后是输出模块，最后一个变换，得到卷积的输出

		self.out = nn.Sequential(
            normalization(ch),
            SiLU(),
            zero_module(conv_nd(dims, model_channels, out_channels, 3, padding=1)),
        )

forward函数，输入x，timesteps，y，y是条件生成

timesteps经过timeset-embedding得到emb表示，这里用的正余弦timeEmbbing，总之，能对不同的timestep实现差异化表示即可

y还得表示成条件的emb

然后对input block的遍历，这个模块的输入是上个模块的输出,中间是middle-block,最后是output-block，之所以有个hs列表，是因为我们得保存input-block的输出，给out-block使用

    def forward(self, x, timesteps, y=None):
        """
        Apply the model to an input batch.

        :param x: an [N x C x ...] Tensor of inputs.
        :param timesteps: a 1-D batch of timesteps.
        :param y: an [N] Tensor of labels, if class-conditional.
        :return: an [N x C x ...] Tensor of outputs.
        """
        assert (y is not None) == (
            self.num_classes is not None
        ), "must specify y if and only if the model is class-conditional"

        hs = []
        emb = self.time_embed(timestep_embedding(timesteps, self.model_channels))

        if self.num_classes is not None:
            assert y.shape == (x.shape[0],)
            emb = emb + self.label_emb(y)

        h = x.type(self.inner_dtype)
        for module in self.input_blocks:
            h = module(h, emb)
            hs.append(h)
        h = self.middle_block(h, emb)
        for module in self.output_blocks:
            cat_in = th.cat([h, hs.pop()], dim=1)
            h = module(cat_in, emb)
        h = h.type(x.dtype)
        return self.out(h)

conv_nd

只是对nn.Conv函数的一个封装

def conv_nd(dims, *args, **kwargs):
    """
    Create a 1D, 2D, or 3D convolution module.
    """
    if dims == 1:
        return nn.Conv1d(*args, **kwargs)
    elif dims == 2:
        return nn.Conv2d(*args, **kwargs)
    elif dims == 3:
        return nn.Conv3d(*args, **kwargs)
    raise ValueError(f"unsupported dimensions: {dims}")

TimestepEmbedSequential emb传入层

TimestepEmbedSequential作用就是对TimestepBlock的子类输入x之外，多传入一个emb

class TimestepEmbedSequential(nn.Sequential, TimestepBlock):
    """
    A sequential module that passes timestep embeddings to the children that
    support it as an extra input.
    """

    def forward(self, x, emb):
        for layer in self:
            if isinstance(layer, TimestepBlock):
                x = layer(x, emb)
            else:
                x = layer(x)
        return x

那么有哪些TimestepBlock的子类呢，只有一个ResBlock类是继承了TimestepBlock，也就是说只有ResBlock才用传入emb，像在上采样和下采样就不用emb了

Downsample 下采样层

下采样层，直接调用了self.op，self.op有卷积的下采样，和直接平均池化的下采样，2d图像中stride=2（3d的stride=（1，2，2）），stride=2作用是对图像空间h,w=1/2h, 1/2w，长和宽减少一半

class Downsample(nn.Module):
    """
    A downsampling layer with an optional convolution.

    :param channels: channels in the inputs and outputs.
    :param use_conv: a bool determining if a convolution is applied.
    :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then
                 downsampling occurs in the inner-two dimensions.
    """

    def __init__(self, channels, use_conv, dims=2):
        super().__init__()
        self.channels = channels
        self.use_conv = use_conv
        self.dims = dims
        stride = 2 if dims != 3 else (1, 2, 2)
        if use_conv:
            self.op = conv_nd(dims, channels, channels, 3, stride=stride, padding=1)
        else:
            self.op = avg_pool_nd(stride)

    def forward(self, x):
        assert x.shape[1] == self.channels
        return self.op(x)

Upsample 上采样层

用临近插值interpolate扩大自己空间h,w两倍，如果要卷积，再做一个通道数不变的上卷积

class Upsample(nn.Module):
    """
    An upsampling layer with an optional convolution.

    :param channels: channels in the inputs and outputs.
    :param use_conv: a bool determining if a convolution is applied.
    :param dims: determines if the signal is 1D, 2D, or 3D. If 3D, then
                 upsampling occurs in the inner-two dimensions.
    """

    def __init__(self, channels, use_conv, dims=2):
        super().__init__()
        self.channels = channels
        self.use_conv = use_conv
        self.dims = dims
        if use_conv:
            self.conv = conv_nd(dims, channels, channels, 3, padding=1)

    def forward(self, x):
        assert x.shape[1] == self.channels
        if self.dims == 3:
            x = F.interpolate(
                x, (x.shape[2], x.shape[3] * 2, x.shape[4] * 2), mode="nearest"
            )
        else:
            x = F.interpolate(x, scale_factor=2, mode="nearest")
        if self.use_conv:
            x = self.conv(x)
        return x

AttentionBlock 注意力机制层

直接看_forward，首先x变成3维的[batch-size,channel,-1]，将x归一化norm，再送入qkv，得到qkv三个量
将qkv reshape，变成 batch-size×num_head，-1（序列长度），qkv.shape[2]（特征维度）
将qkv送如QKVAttention类，得到h，h是经过注意力之后的结果，将h reshape，再经过投影层，加回x，所以这是一个带残差的attention注意力机制

class AttentionBlock(nn.Module):
    """
    An attention block that allows spatial positions to attend to each other.

    Originally ported from here, but adapted to the N-d case.
    https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/diffusion_tf/models/unet.py#L66.
    """

    def __init__(self, channels, num_heads=1, use_checkpoint=False):
        super().__init__()
        self.channels = channels
        self.num_heads = num_heads
        self.use_checkpoint = use_checkpoint

        self.norm = normalization(channels)
        self.qkv = conv_nd(1, channels, channels * 3, 1)
        self.attention = QKVAttention()
        self.proj_out = zero_module(conv_nd(1, channels, channels, 1))

    def forward(self, x):
        return checkpoint(self._forward, (x,), self.parameters(), self.use_checkpoint)

    def _forward(self, x):
        b, c, *spatial = x.shape
        x = x.reshape(b, c, -1)
        qkv = self.qkv(self.norm(x))
        qkv = qkv.reshape(b * self.num_heads, -1, qkv.shape[2])
        h = self.attention(qkv)
        h = h.reshape(b, -1, h.shape[-1])
        h = self.proj_out(h)
        return (x + h).reshape(b, c, *spatial)

QKVAttention

这个就是标准的attention计算

class QKVAttention(nn.Module):
    """
    A module which performs QKV attention.
    """

    def forward(self, qkv):
        """
        Apply QKV attention.

        :param qkv: an [N x (C * 3) x T] tensor of Qs, Ks, and Vs.
        :return: an [N x C x T] tensor after attention.
        """
        ch = qkv.shape[1] // 3
        q, k, v = th.split(qkv, ch, dim=1)
        scale = 1 / math.sqrt(math.sqrt(ch))
        weight = th.einsum(
            "bct,bcs->bts", q * scale, k * scale
        )  # More stable with f16 than dividing afterwards
        weight = th.softmax(weight.float(), dim=-1).type(weight.dtype)
        return th.einsum("bts,bcs->bct", weight, v)

    @staticmethod
    def count_flops(model, _x, y):
        """
        A counter for the `thop` package to count the operations in an
        attention operation.

        Meant to be used like:

            macs, params = thop.profile(
                model,
                inputs=(inputs, timestamps),
                custom_ops={QKVAttention: QKVAttention.count_flops},
            )

        """
        b, c, *spatial = y[0].shape
        num_spatial = int(np.prod(spatial))
        # We perform two matmuls with the same number of ops.
        # The first computes the weight matrix, the second computes
        # the combination of the value vectors.
        matmul_ops = 2 * b * (num_spatial ** 2) * c
        model.total_ops += th.DoubleTensor([matmul_ops])

ResBlock

有很多层，有in_layer层，emb-layer层，out-layers层，还有skip-connection层，如果通道数一致则直接连接起来就好，如果通道数目不一致，可以用一个大小不变的卷积或者1×1的卷积改变一下dim

        self.in_layers = nn.Sequential(
            normalization(channels),
            SiLU(),
            conv_nd(dims, channels, self.out_channels, 3, padding=1),
        )
        self.emb_layers = nn.Sequential(
            SiLU(),
            linear(
                emb_channels,
                2 * self.out_channels if use_scale_shift_norm else self.out_channels,
            ),
        )
        self.out_layers = nn.Sequential(
            normalization(self.out_channels),
            SiLU(),
            nn.Dropout(p=dropout),
            zero_module(
                conv_nd(dims, self.out_channels, self.out_channels, 3, padding=1)
            ),
        )

        if self.out_channels == channels:
            self.skip_connection = nn.Identity()
        elif use_conv:
            self.skip_connection = conv_nd(
                dims, channels, self.out_channels, 3, padding=1
            )
        else:
            self.skip_connection = conv_nd(dims, channels, self.out_channels, 1)

forward函数，传入x和emb，x经过in_layers得到h，emb经过emb_layers得到emb-out，h + emb_out输入out_layers得到h，x再和h相加，所以大致就是x和h的一个残差连接

    def _forward(self, x, emb):
        h = self.in_layers(x)
        emb_out = self.emb_layers(emb).type(h.dtype)
        while len(emb_out.shape) < len(h.shape):
            emb_out = emb_out[..., None]
        if self.use_scale_shift_norm:
            out_norm, out_rest = self.out_layers[0], self.out_layers[1:]
            scale, shift = th.chunk(emb_out, 2, dim=1)
            h = out_norm(h) * (1 + scale) + shift
            h = out_rest(h)
        else:
            h = h + emb_out
            h = self.out_layers(h)
        return self.skip_connection(x) + h

写在后面

其实实现的思路很简单，只是要把它写成模块就稍显的有些复杂了，这就是我们可以多多学习的地方，有时间仿照着这个写一下…