CrossEntropy的numpy实现和Pytorch调用

猛男炸鸡翅

已于 2022-03-30 16:04:10 修改

阅读量5.7k

点赞数 4

分类专栏： Coding 文章标签：深度学习 python

于 2020-07-31 16:08:17 首次发布

本文链接：https://blog.youkuaiyun.com/weixin_43508499/article/details/107714080

版权

Coding 专栏收录该内容

18 篇文章

订阅专栏

本文详细介绍了如何使用Numpy和Pytorch两种不同的方法来实现交叉熵损失函数，包括基本的实现过程和使用Pytorch的nn.CrossEntropyLoss模块的高级应用，特别关注了weight参数在解决样本不平衡问题中的作用。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1. Numpy实现

import torch
import numpy as np
from torch.nn import functional as F


# 定义softmax函数
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x))

# 利用numpy计算
def cross_entropy_np(x, y):
    x_softmax = [softmax(x[i]) for i in range(len(x))]
    x_log = [np.log(x_softmax[i][y[i]]) for i in range(len(y))]
    loss = - np.sum(x_log) / len(y)
    return loss


# 另外一种实现方式，y转成onehot形式，比较直观
def cross_entropy_np2(x, y):
    num_data, num_class = x.shape
    log_p = np.array([np.log(softmax(x[i])) for i in range(num_data)])
    y_onehot = np.eye(num_class)[y]
    loss = - np.sum(y_onehot * log_p) / num_data
    return round(loss, 4)

if __name__ == '__main__':
    # 假设有数据x, y
    x = np.array([[0.093, 0.1939, -1.0649, 0.4476, -2.0769],
                  [-1.8024, 0.3696, 0.7796, -1.0346, 0.473],
                  [0.5593, -2.5067, -2.1275, 0.5548, -1.6639]])
    
    y = np.array([1, 2, 3])
    print('numpy result: ', cross_entropy_np(x, y)) 
# numpy result:  1.0155949508195155

2. Pytorch实现

import torch
import numpy as np
from torch.nn import functional as F


# 调用Pytorch的nn.CrossEntropy计算
def cross_entropy_pth(x, y):
    x_pth = torch.from_numpy(x)
    y_pth = torch.from_numpy(y).long()
    loss = F.cross_entropy(x_pth, y_pth)
    return loss

if __name__ == '__main__':
    # 假设有数据x, y
    x = np.array([[0.093, 0.1939, -1.0649, 0.4476, -2.0769],
                  [-1.8024, 0.3696, 0.7796, -1.0346, 0.473],
                  [0.5593, -2.5067, -2.1275, 0.5548, -1.6639]])
    
    y = np.array([1, 2, 3])
    print('pytorch result: ', cross_entropy_pth(x, y))
# pytorch result:  tensor(1.0156, dtype=torch.float64)

3. nn.CrossEntropy的weight参数

下图是CrossEntropy的说明文档：
在这里插入图片描述 weight参数给不同类别项的loss分配了不同权重，可以解决样本不均衡问题.

可以看到weight的Size必须是跟所分的类别一样长，如上述代码中所示，每个x数据有5维数据（即文档中的C），表示有5个类别，因此weight的维度必须是5

以下举例说明：

import torch
import numpy as np
from torch.nn import functional as F


# 调用Pytorch的nn.CrossEntropy计算
def cross_entropy_pth(x, y, weight=None):
    x_pth = torch.from_numpy(x)
    y_pth = torch.from_numpy(y).long()
    weight = torch.from_numpy(weight).double()
    loss = F.cross_entropy(x_pth, y_pth, weight=weight)
    return loss


# numpy定义softmax函数
def softmax(x):
    return np.exp(x) / np.sum(np.exp(x))


# 利用numpy计算
def cross_entropy_np(x, y, weight=None):
    x_softmax = [softmax(x[i]) for i in range(len(x))]
    x_log = [np.log(x_softmax[i][y[i]]) for i in range(len(y))]
    if len(weight):
        x_log = [x_log[i]*weight[y[i]] for i in range(len(y))]
        total_num = sum([weight[i] for i in y])
    else:
        total_num = len(y)
    loss = - np.sum(x_log) / total_num
    return loss


if __name__ == '__main__':
    # 假设有数据x, y
    x = np.array([[0.093, 0.1939, -1.0649, 0.4476, -2.0769],
                  [-1.8024, 0.3696, 0.7796, -1.0346, 0.473],
                  [0.5593, -2.5067, -2.1275, 0.5548, -1.6639]])
    
    y = np.array([1, 2, 3])
    
    #  假设给第2，3类分配更多权重，权重参数设为2
    weight = np.array([1, 1, 2, 2, 1])
    
    print('numpy result: ', cross_entropy_np(x, y, weight))
    print('pytorch result: ', cross_entropy_pth(x, y, weight=weight))
# numpy result:  0.9636395529496566
# pytorch result:  tensor(0.9636, dtype=torch.float64)