Normalization Techniques: BN and LN

Normalization Techniques

Motivation: Why needs Normalization?

Normalization has always been an active area of research in deep learning. There are some benefits of it:

• It reduces Internal Covariate Shift. It is the change in the distribution of network activations due to the change in network parameters during training. It will move many of the distribution of network activations into the saturated regime of the nonlinearity and slow down the convergence. To improve training, we seek to reduce the internal covariate shift.
• Normalization makes loss surface smoother.
• It makes the Optimization faster because normalization doesn’t allow weights to explode all over the place and restricts them to a certain range.

Batch Normalization

Training Phase

Batch normalization is a method that normalizes activations in a network across the mini-batch of definite size. For each feature, batch normalization computes the mean and variance of that feature in the mini-batch . It then substracts the mean and divides the feature by its mini-batch standard deviation.

However, simply normalizing each input of a layer may change what the layer can represent. For instance, normalizing the inputs of a sigmoid would constrain them to the linear regime of the nonlinearity. To address this and make sure the transformation inserted in the network can represent the identity transform, a pair of parameters ${\gamma }^{(k)},{\beta }^{(k)}$ where introduced to scale and shift the normalized value for each activation $x^{(k)}$:
$$y^{(k)}={\gamma }^{(k)}{\hat{x}}^{(k)}+{\beta }^{(k)}$$
These parameters are learned along with the original model parameters, and restore the representation power of the network. Indeed, by setting ${\gamma }^{(k)}=\sqrt{Var[x^{(k)}]}$ and ${\beta }^{(k)}=E[x^{(k)}]$, we could recover the original activatioins, if that were the optimal thing to do.

The batch normalization can be summarized as:

Inference Phase

During the inference phase, specifically when you only have one test sample, doing batch normalization as the one in training phase does not make sense, because your outputs at each layer of the network will be exactly zero. Instead. we use running mean and running variance calculated during training.

At each time step $t$ during training, the runing mean ${\mu }_B^{\prime }[t]$ and running variance ${\sigma }_B^{\prime 2}$ are calculated as:
$${\mu }_B^{\prime }[t]={\mu }_B^{\prime }[t]\times momentum+{\mu }_B[t]\times (1-momentum),$$

$${\sigma }_B^{\prime 2}[t]={\sigma }_B^{\prime 2}[t]\times momentum+{\sigma }_B^2[t]\times (1-momentum).$$

Layer Normalization

Given inputs $x$ over a minibatch of size $m$, B = { x 1 , x 2 , . . . , x m } B=\{x_1,x_2,...,x_m\} B={x1​,x2​,...,xm​}, each sample $x_i$ contains $K$ elements, i.e. the length of flatten $x_i$ is $K$, by applying transformation of your inputs using some learned parameters $\gamma$ and $\beta$, the outputs could be expressed as B ′ = { y 1 , y 2 , . . . , y m } B'=\{y_1,y_2,...,y_m\} B′={y1​,y2​,...,ym​}, where $y_i=L{N}_{\gamma ,\beta }(x_i)$.

More concretely, we first calculate the mean and the variance of each sample from the minibatch. For sample $x_i$ whose flatten format is { x i , 1 , x i , 2 , . . . , x i , K } \{x_{i,1},x_{i,2},...,x_{i,K}\} {xi,1​,xi,2​,...,xi,K​}, we have its mean ${\mu }_i$ and variance ${\sigma }_i^2$.
$${\mu }_i=\frac{1}{K}\sum _{k=1}^K{x}_{i,k},$$

$${\sigma }_i^2=\frac{1}{K}\sum _{k=1}^K(x_{i,k}-{\mu }_i{)}^2.$$

Then we normalize each sample such that the elements in the sample have zero mean and unit variance. $ϵ$ is for numerical stability in case the denominator becomes zero by chance:
$${\hat{x}}_{i,k}=\frac{x_{i,k}-{\mu }_i}{\sqrt{{\sigma }_i^2+ϵ}}.$$
Finally, there is a scaling and shifting step like in batch normalization. $\gamma$ and $\beta$ are learnable parameters.
$$y_i=\gamma {\hat{x}}_i+\beta \equiv L{N}_{\gamma ,\beta }(x_i).$$
Layer normalization has nothing to do with other sample in a mini batch.

Layer normalization for CNN

For CNN, each channel is considered as an “independent” sample and all the normalization was done for that specific channel only within the sample.

Assume the input tensor has shape $[b,H,W,C]$, for each channel c ∈ { 1 , 2 , . . . , C } c\in \{1,2,...,C\} c∈{1,2,...,C}
$${\mu }_{i,c}=\frac{1}{HW}\sum _{j=1}^H\sum _{k=1}^H{x}_{i,j,k,c},$$

$${\sigma }_{i,c}^2=\frac{1}{HW}\sum _{j=1}^H\sum _{k=1}^H(x_{i,j,k,c}-{\mu }_{i,c}{)}^2,$$

$${\hat{x}}_{i,j,k,c}=\frac{x_{i,j,k,c}-{\mu }_{i,c}}{\sqrt{{\sigma }_{i,c}^2+ϵ}}.$$

Similarly, we have learnable parameters ${\gamma }_c$ and ${\beta }_c$ for each channel.

References

• Layer Normalization Explained
• Normalization Techniques in Deep Neural Networks