FLOPS计算

最新推荐文章于 2025-04-30 16:26:05 发布

原创

最新推荐文章于 2025-04-30 16:26:05 发布 · 303 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#大模型 #GPU

训练的FLOPS:

Llama2 (DGXC Benchmarking) | NVIDIA NGC

The calculations for the 7b parameter model:

model flops = (sequence length) * ((attention flops) + (mlp flops) + (embedding flops))

model flops breakdown:
    attention flops = 12 * (number of layers) * (hidden size)^2 * (1 + (number of query groups)/(number of attention heads) + (sequence length)/(hidden size))
    mlp flops = 18 * (number of layers) * (FFN size) * (hidden size)
    embedding flops = 6 * (vocab size) * (hidden size)

Llama 2 7b calculation:
    sequence length = 4096
    attention flops = 12 * 32 * 4096^2 * (1 + 32/32 + 4096/4096) = 19,327,352,832
    mlp flops = 18 * 32 * 11008 * 4096 = 25,971,130,368
    embedding flops = 6 * 32000 * 4096 = 786,432,000

    model flops = 4096 * (19,327,