original Residual Unit && full pre-activation Residual Unit

最新推荐文章于 2021-04-03 19:11:01 发布

原创最新推荐文章于 2021-04-03 19:11:01 发布 · 2.5k 阅读

2 ·

CC 4.0 BY-SA版权

文章标签：

#ResNet

深度学习专栏收录该内容

21 篇文章

订阅专栏

本文详细介绍了2016年ECCV上提出的full预激活ResNet结构，并通过对比原始ResNet164结构，展示了其如何通过改进降低错误率0.5%。文中提供了PyTorch代码实现，对比了两种ResNet单元的forward函数差异。

在2016年ECCV的一篇论文中，讲述到了full pre-activation ResNet。

其改进的ResNet164结构比original结构的error降低0.5%

K. He, X. Zhang, S. Ren, and J. Sun, “Identity mapping in deep residual networks,” in ECCV 2016
在这里插入图片描述

图中的weight层代表的就是卷积层操作，BN和relu分别是批正则化和激活函数

original Residual Unit Pytorch实现

# full pre-activation  Residual Unit
class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, in_channel, out_channel, stride=1, downsample=None, norm_layer=None):
        super(Bottleneck, self).__init__()
        if norm_layer is None:
            norm_layer = nn.BatchNorm2d   # BN层

        self.conv1 = nn.Conv2d(in_channels=in_channel, out_channels=out_channel,
                               kernel_size=1, stride=1, bias=False)  # squeeze channels
        self.bn1 = norm_layer(out_channel)
        # -----------------------------------------
        self.conv2 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel,
                               kernel_size=3, stride=stride, bias=False, padding=1)
        self.bn2 = norm_layer(out_channel)
        # -----------------------------------------
        self.conv3 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel * self.expansion,
                               kernel_size=1, stride=1, bias=False)  # unsqueeze channels
        self.bn3 = norm_layer(out_channel * self.expansion)
        self.relu = nn.ReLU(inplace=True)  # relu激活函数
        self.downsample = downsample

    def forward(self, x):
        identity = x
        if self.downsample is not None:
            identity = self.downsample(x)
		out = self.bn1(x)
		out = self.relu(out)
        out = self.conv1(out)
       
		out = self.bn2(out)
        out = self.relu(out)
        out = self.conv2(out)
        
		out = self.bn3(out)
		out = self.relu(out)
        out = self.conv3(out)

        out += identity

        return out

两者的主要区别在于forward函数的区别，init函数的定义部分没有区别。

full pre-activation Residual Unit Pytorch代码实现

# 最原始的ResNet Block
class Bottleneck(nn.Module):
    expansion = 4

    def __init__(self, in_channel, out_channel, stride=1, downsample=None, norm_layer=None):
        super(Bottleneck, self).__init__()
        if norm_layer is None:
            norm_layer = nn.BatchNorm2d  # BN层

        self.conv1 = nn.Conv2d(in_channels=in_channel, out_channels=out_channel,
                               kernel_size=1, stride=1, bias=False)  # squeeze channels
        self.bn1 = norm_layer(out_channel)
        # -----------------------------------------
        self.conv2 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel,
                               kernel_size=3, stride=stride, bias=False, padding=1)
        self.bn2 = norm_layer(out_channel)
        # -----------------------------------------
        self.conv3 = nn.Conv2d(in_channels=out_channel, out_channels=out_channel * self.expansion,
                               kernel_size=1, stride=1, bias=False)  # unsqueeze channels
        self.bn3 = norm_layer(out_channel * self.expansion)
        self.relu = nn.ReLU(inplace=True)  # relu激活函数
        self.downsample = downsample

    def forward(self, x):
        identity = x
        if self.downsample is not None:
            identity = self.downsample(x)

        out = self.conv1(x)
        out = self.bn1(out)
        out = self.relu(out)

        out = self.conv2(out)
        out = self.bn2(out)
        out = self.relu(out)

        out = self.conv3(out)
        out = self.bn3(out)

        out += identity
        out = self.relu(out)

        return out