LoRA 和 DoRA 代码笔记

最新推荐文章于 2025-05-27 19:46:18 发布

Qiming_v

最新推荐文章于 2025-05-27 19:46:18 发布

阅读量558

点赞数 9

文章标签： pytorch LoRA DoRA

本文链接：https://blog.youkuaiyun.com/zhilaizhiwang/article/details/141755192

版权

Improving LoRA: Implementing Weight-Decomposed Low-Rank Adaptation (DoRA) from Scratch

LoRA

在这里插入图片描述
LoRA初始化时，A使用正态分布，B使用0.

class LoRALayer(nn.Module):
    def __init__(self, in_dim, out_dim, rank, alpha):
        super().__init__()
        std_dev = 1 / torch.sqrt(torch.tensor(rank).float())
        self.A = nn.Parameter(torch.randn(in_dim, rank) * std_dev)
        self.B = nn.Parameter(torch.zeros(rank, out_dim))
        self.alpha = alpha

    def forward(self, x):
        x = self.alpha * (x @ self.A @ self.B