Group Norm 学习笔记

本文深入探讨了GroupNormalization(GroupNorm)技术,该方法在小批量训练时展现出显著优势,与BatchNorm、LayerNorm和InstanceNorm相比。GroupNorm通过将通道分组并在每个组内进行归一化,解决了BatchNorm在小批量数据上的问题。提供的代码展示了GroupNorm的实现过程,实验证明其在各种场景下都能有效稳定模型训练。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

论文:《Group Normalization》
地址https://arxiv.org/pdf/1803.08494.pdf
优势:小批量训练时优势明显(相比另三个 Norm)
其他Batch Norm、Layer Norm、Instance Norm

在这里插入图片描述

方法

将通道分组,在每组做 Norm

代码

源自论文

def GroupNorm(x, gamma, beta, G, eps=1e5):
	# x: input features with shape [N,C,H,W]
	# gamma, beta: scale and offset, with shape [1,C,1,1] 
	# G: number of groups for GN
	N, C, H, W = x.shape
	x = tf.reshape(x, [N, G, C // G, H, W])
	
	mean, var = tf.nn.moments(x, [2, 3, 4], keep dims=True) 
	x = (x − mean) / tf.sqrt(var + eps)
	
	x = tf.reshape(x, [N, C, H, W]) 
	
	return x ∗ gamma + beta
实验

在这里插入图片描述
在这里插入图片描述
在这里插入图片描述

### Group Normalization in Neural Networks #### Definition of GroupNorm Group normalization (GN) is a technique used to normalize the features within each group across channels during training deep neural networks. Unlike batch normalization which normalizes over batches, GN divides channel dimensions into groups and then computes statistics such as mean and variance inside each individual group[^1]. This approach makes GN less sensitive to batch size compared to other methods like Batch Norm. #### Usage Scenarios for GroupNorm In scenarios where small or variable-sized batches are common, including tasks involving sequential data processing or when working with limited hardware resources that restrict batch sizes, GN can be particularly advantageous. By reducing dependence on large batch sizes while maintaining performance comparable to BN under certain conditions, GN offers flexibility without compromising accuracy significantly[^2]. #### Implementation Details To implement GroupNorm effectively: ```python import torch.nn as nn class MyModel(nn.Module): def __init__(self): super(MyModel, self).__init__() # Assume input has C=64 channels; divide them into G=8 groups. self.group_norm = nn.GroupNorm(num_groups=8, num_channels=64) def forward(self, x): out = self.group_norm(x) return out ``` This code snippet demonstrates how easily `nn.GroupNorm` from PyTorch library integrates into custom models. Here, an instance named `group_norm` is created specifying both number of groups (`num_groups`) and corresponding total count of feature maps/channels(`num_channels`). During inference phase through method overriding `forward`, tensor `x` gets normalized according to defined parameters before being returned as output.
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值