Pytorch（二） —— 激活函数、损失函数及其梯度

最新推荐文章于 2024-03-22 18:29:24 发布

CyrusMay

最新推荐文章于 2024-03-22 18:29:24 发布

阅读量855

点赞数 4

CC 4.0 BY-SA版权

分类专栏： Pytorch 深度学习（神经网络）专题文章标签： pytorch 机器学习深度学习人工智能 python

本文链接：https://blog.youkuaiyun.com/Cyrus_May/article/details/125500584

深度学习（神经网络）专题同时被 2 个专栏收录

46 篇文章

订阅专栏

Pytorch

7 篇文章

订阅专栏

本文介绍了PyTorch中常用的激活函数，包括Sigmoid、Tanh、ReLU和Softmax，以及它们的导数。接着讲解了MSE和CrossEntropy两种损失函数，并通过示例展示了如何使用PyTorch实现。最后，探讨了求导和反向传播的过程，演示了如何计算梯度并应用反向传播更新权重。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

1.激活函数

1.1 Sigmoid / Logistic

$\delta(x)=\frac{1}{1+e^{-x}}\\\delta'(x)=\delta(1-\delta)$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.sigmoid(x)
plt.plot(x,y)
plt.show()

在这里插入图片描述

1.2 Tanh

$tanh(x)=\frac{e^x-e^{-x}}{e^x+e^{-x}}\\\frac{\partial tanh(x)}{\partial x}=1-tanh^2(x)$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.tanh(x)
plt.plot(x,y)
plt.show()

在这里插入图片描述

1.3 ReLU

$f (x) = m a x (0, x)$

import matplotlib.pyplot as plt
import torch.nn.functional as F
x = torch.linspace(-10,10,1000)
y = F.relu(x)
plt.plot(x,y)
plt.show()

在这里插入图片描述

1.4 Softmax

$p_i=\frac{e^{a_i}}{\sum_{k=1}^N{e^{a_k}}}\\ \frac{\partial p_i}{\partial a_j}=\left\{ \begin{array}{lc} p_i(1-p_j) & i=j \\ -p_ip_j&i\neq j\\ \end{array} \right.$

import torch.nn.functional as F
logits = torch.rand(10)
prob = F.softmax(logits,dim=0)
print(prob)

tensor([0.1024, 0.0617, 0.1133, 0.1544, 0.1184, 0.0735, 0.0590, 0.1036, 0.0861,
        0.1275])

2.损失函数

2.1 MSE

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
mse = F.mse_loss(y,x@w)
print(mse)

tensor(238.5115)

2.2 CorssEntorpy

import torch.nn.functional as F
x = torch.rand(100,64)
w = torch.rand(64,10)
y = torch.randint(0,9,[100])
entropy = F.cross_entropy(x@w,y)
print(entropy)

tensor(3.6413)

3. 求导和反向传播

3.1 求导

Tensor.requires_grad_()
torch.autograd.grad()

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,1)
y = torch.rand(100,1)
w.requires_grad_()
mse = F.mse_loss(x@w,y)
grads = torch.autograd.grad(mse,[w])
print(grads[0].shape)

torch.Size([64, 1])

3.2 反向传播

Tensor.backward()

import torch.nn.functional as F
import torch
x = torch.rand(100,64)
w = torch.rand(64,10)
w.requires_grad_()
y = torch.randint(0,9,[100,])
entropy = F.cross_entropy(x@w,y)
entropy.backward()
w.grad.shape