DeepLearning: Conv/Deconv/Relu/Loss/BatchNorm 的关系（非理论探究）

Contact Me:
王雪豪 xuehaowang@buaa.edu.cn

It is generally known that the order is important when you use the Conv Layers and Activation Layers.

We usually use them as follow:

conv -> relu
conv -> batchnorm -> scale -> relu

This order shown above can work well. However, how to add into a Deconv Layer?

We use Deconv Layers to upsample the input data and the paras can be set as this(caffe framwork):

layer {
  name: "deconv"
  type: "Deconvolution"
  bottom: "conv"
  top: "deconv"
  param {lr_mult: 0 decay_mult: 0}
  param {lr_mult: 0 decay_mult: 0}
  convolution_param {
    num_output: 1
    pad: 4
    kernel_size: 16
    stride: 8 #扩大的倍数
    weight_filler {
      type:"bilinear" #差值填充
    }
    bias_filler {
      type: "constant"
      value: 0
    }
  }
}

This is a part of caffe network designed to upsample the input 'conv'. The lr is set as 0 to avoid changing the paras during training phase.

So if we want to combine the Conv layer and Deconv layer, we could follow this order:

Conv -> Deconv -> Relu
Conv -> Deconv -> batchnorm -> scale -> Relu

Essentially the Deconv layer can be treat as a post-operate of the Conv layer when you just want to upsample input data and Do Not Need a Activation Layer to active the Conv layer additionally.

Meanwhile if your network has a Loss layer to deal with the data from Conv and Deconv, you must directly connect the loss layer with Deconv layer instead of Conv layer, unless your data from Deconv and Conv layer are both single channel.

The follow order can work well and if you conect the loss layer with conv layer which is output by the deconv layer, you will get an unstable loss value.

Conv(维度下降） -> Deconv（上采样） -> loss
INSTEAD OF
Deconv（上采样） -> Conv（维度下降） -> loss