【Llama源码】归一化RMSNorm

数学公式与代码

RMSNorm是在Layer Norm之上的改进,它通过舍弃中心不变性来降低计算量。

a ‾ i = a i R M S ( a ) g i 其中, R M S ( a ) = 1 n ∑ r = 1 n a i 2 \overline a_i = \frac {a_i}{RMS(a)} g_i \\ 其中,RMS(a)=\sqrt { { \frac1n}{\sum_{r=1}^n a_i^2}} ai=RMS(a)aig

03-08
### RMSNorm in Machine Learning #### Definition of RMSNorm Root Mean Square Normalization (RMSNorm) is a method used to normalize the output activations of neural network layers. Unlike Layer Normalization which normalizes based on both mean and variance, RMSNorm only uses the root mean square statistic without centering by subtracting the mean[^1]. This simplification can lead to faster computation while still providing effective normalization. The formula for computing RMSNorm over an input vector \( \mathbf{x} \) with dimensionality D is as follows: \[ y_i = \frac{x_i}{(\frac{1}{D}\sum_{d=1}^{D}(x_d^2))^{\frac{1}{2}}+\epsilon} \] where \( y_i \) represents each normalized element from the original tensor \( x_i \), and \( \epsilon \) is added for numerical stability during division operations. #### Implementation Example Below demonstrates how one might implement RMSNorm within PyTorch framework using Python code snippet. ```python import torch from torch import nn class RMSNorm(nn.Module): def __init__(self, dim: int, eps: float = 1e-8): super().__init__() self.eps = eps self.scale = nn.Parameter(torch.ones(dim)) def forward(self, x): norm_x = x / ((torch.mean(x ** 2, dim=-1, keepdim=True)) + self.eps).sqrt() return self.scale * norm_x ``` This class defines `__init__` function initializing parameters including epsilon value ensuring no divide-by-zero errors occur when calculating norms; also initializes scaling parameter per feature channel. The `forward` method computes squared means across last axis (-1 indicates batch dimensions are preserved), adds small constant term before taking square roots thus avoiding potential instability issues due to very low values close zero after addition operation completes successfully.
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值