pytorch中torch.nn.functional.normalize、nn.norm和LayerNorm计算中的区别

torch.nn.functional.normalize

计算方式

$$\mathbf{v}=\frac{\mathbf{v}}{\max \left(\Vert \mathbf{v}{\Vert }_p,ϵ\right)}$$

pytorch例子

a = torch.arange(12).reshape(3,4).float()
torch.nn.functional.normalize(a, dim=-1)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],
        [0.3563, 0.4454, 0.5345, 0.6236],
        [0.4182, 0.4704, 0.5227, 0.5750]])
'''

这样除一下就相等了

a = torch.arange(12).reshape(3,4).float()
torch.nn.functional.normalize(a, dim=-1)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],
        [0.3563, 0.4454, 0.5345, 0.6236],
        [0.4182, 0.4704, 0.5227, 0.5750]])
'''
a / a.norm(dim=-1,keepdim=True)
'''
tensor([[0.0000, 0.2673, 0.5345, 0.8018],
        [0.3563, 0.4454, 0.5345, 0.6236],
        [0.4182, 0.4704, 0.5227, 0.5750]])
'''

nn.norm

计算方式

可以通过p参数指定是pro(Frobenium norm，其实就是平方和开根号)还是nuc（叫nuclear norm，其实就是SVD分解的奇异值，也就是Singular Value）

pytorch例子

a = torch.arange(12).reshape(3,4).float()
a.norm(dim=-1, keepdim=True)
'''
tensor([[ 3.7417],
        [11.2250],
        [19.1311]])
'''

LayerNorm

计算方式

分子减均值，分母对方差开根号，更多细节参考 超细节的BatchNorm/BN/LayerNorm/LN/SyncLN/ShuffleBN/RMSNorm知识点

pytorch例子

a = torch.arange(12).reshape(3,4).float()
(a - a.mean(dim=-1, keepdims=True)) / torch.sqrt(a.var(dim=-1, keepdims=True, correction=0))
'''
tensor([[-1.3416, -0.4472,  0.4472,  1.3416],
        [-1.3416, -0.4472,  0.4472,  1.3416],
        [-1.3416, -0.4472,  0.4472,  1.3416]])
'''