Transformer 中的 Positional Encoding 不同版本的实现_transfoemermodel positionalencoding python-优快云博客

该博客主要展示了如何使用Python的numpy和pytorch库来实现Transformer模型中的位置编码（Positional Encoding）。通过数学公式，分别用sin和cos函数生成位置向量，并用图像展示其分布。内容涉及矩阵运算和图像可视化。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.根据原文公式实现

import numpy as np
import math
 
import matplotlib.pyplot as plt

max_len = 10  # 单词个数
d_model = 16   # 位置向量维度大小

pe = np.zeros((max_len, d_model))

for pos in range(max_len):
    for i in range(d_model // 2):
        pe[pos, 2 * i] = np.sin(pos / np.power(10000, 2 * i / d_model))
        pe[pos, 2 * i + 1] = np.cos(pos / np.power(10000, 2 * i / d_model))



plt.imshow(pe)  # 显示图片
plt.colorbar()  # 显示颜色渐变条
plt.axis('off') # 不显示坐标轴
plt.show()

2.根据公式推导实现

https://blog.youkuaiyun.com/qq_15534667/article/details/116140592

numpy

import numpy as np
import math
import matplotlib.pyplot as plt

max_len = 10  # 单词个数
d_model = 16   # 位置向量维度大小

pe = np.zeros((max_len, d_model))


position = np.arange(0., max_len) 
position = np.reshape(position, [max_len,1])

 
div_term = np.exp(np.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))

pe[:, 0::2] = np.sin(position * div_term)    # 偶数列
pe[:, 1::2] = np.cos(position * div_term)    # 奇数列

plt.imshow(pe)  # 显示图片
plt.colorbar()  # 显示颜色渐变条
plt.axis('off') # 不显示坐标轴
plt.show()

pytorch

import matplotlib.pyplot as plt

import random
import time
from pathlib import Path

import numpy as np
import torch
import math
 
max_len = 10  # 单词个数
d_model = 16   # 位置向量维度大小


pe = torch.zeros(max_len, d_model)
print pe

position = torch.arange(0., max_len).unsqueeze(1)
print position

div_term = torch.exp(torch.arange(0., d_model, 2) * -(math.log(10000.0) / d_model))
 
print div_term

 
pe[:, 0::2] = torch.sin(position * div_term)    # 偶数列
pe[:, 1::2] = torch.cos(position * div_term)    # 奇数列

pe = pe.unsqueeze(0) 

print pe

pe = pe.numpy()


pe = pe.squeeze()
print pe 

 

plt.imshow(pe) # 显示图片
plt.colorbar()

plt.axis('off') # 不显示坐标轴
plt.show()

原理参考

https://wmathor.com/index.php/archives/1453/