PyTorch-Tutorials【pytorch官方教程中英文详解】- 7 Optimization

最新推荐文章于 2025-07-08 23:51:28 发布

原创最新推荐文章于 2025-07-08 23:51:28 发布 · 999 阅读

1 ·

CC 4.0 BY-SA版权

文章标签：

#pytorch #人工智能 #python #机器学习 #深度学习

笔记同时被 3 个专栏收录

165 篇文章

订阅专栏

72 篇文章

订阅专栏

DL框架

28 篇文章

订阅专栏

本文介绍如何选择损失函数和优化器来优化模型参数。通过设置超参数，使用优化循环进行模型训练，并评估模型性能。

在上一篇文章PyTorch-Tutorials【pytorch官方教程中英文详解】- 6 Autograd介绍了torch.autograd，接下来看看在模型的训练和优化中如何选取损失函数和优化器。

原文链接：Optimizing Model Parameters — PyTorch Tutorials 1.10.1+cu102 documentation

OPTIMIZING MODEL PARAMETERS
Now that we have a model and data it’s time to train, validate and test our model by optimizing its parameters on our data. Training a model is an iterative process; in each iteration (called an epoch) the model makes a guess about the output, calculates the error in its guess (loss), collects the derivatives of the error with respect to its parameters (as we saw in the previous section), and optimizes these parameters using gradient descent. For a more detailed walkthrough of this process, check out this video on backpropagation from 3Blue1Brown.

【现在我们有了一个模型和数据，是时候通过在数据上优化其参数来训练、验证和测试我们的模型了。训练一个模型是一个迭代的过程；在每次迭代(称为epoch)中，模型对输出进行猜测，计算猜测中的误差(损失)，收集误差对其参数的导数(正如我们在前一节中看到的)，并使用梯度下降优化这些参数。关于这个过程的更详细的演练，请查看来自backpropagation from 3Blue1Brown。】

1 Prerequisite Code

We load the code from the previous sections on Datasets & DataLoaders and Build Model.

【我们将从前面的数据集和数据载入器以及构建模型章节中加载代码。】

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets
from torchvision.transforms import ToTensor, Lambda

training_data = datasets.FashionMNIST(
    root="data",
    train=True,
    download=True,
    transform=ToTensor()
)

test_data = datasets.FashionMNIST(
    root="data",
    train=False,
    download=True,
    transform=ToTensor()
)

train_dataloader = DataLoader(training_data, batch_size=64)
test_dataloader = DataLoader(test_data, batch_size=64)

class NeuralNetwork(nn.Module):
    def __init__(self):
        super(NeuralNetwork, self).__init__()
        self.flatten = nn.Flatten()
        self.linear_relu_stack = nn.Sequential(
            nn.Linear(28*28, 512),
            nn.ReLU(),
            nn.Linear(512, 512),
            nn.ReLU(),
            nn.Linear(512, 10),
        )

    def forward(self, x):
        x = self.flatten(x)
        logits = self.linear_relu_stack(x)
        return logits

model = NeuralNetwork()

2 Hyperparameters

Hyperparameters are adjustable parameters that let you control the model optimization process. Different hyperparameter values can impact model training and convergence rates (read more about hyperparameter tuning)

【超参数是可调节的参数，可以让您控制模型优化过程。不同的超参数值会影响模型训练和收敛速度(阅读更多关于超参数调优的信息)】

We define the following hyperparameters for training:

【我们为训练定义了以下超参数:】

Number of Epochs - the number times to iterate over the dataset
【epoch的数量-在数据集上迭代的次数】
Batch Size - the number of data samples propagated through the network before the parameters are updated
【批量大小—在参数更新之前通过网络传播的数据样本的数量】
Learning Rate - how much to update models parameters at each batch/epoch. Smaller values yield slow learning speed, while large values may result in unpredictable behavior during training.
【学习率-在每个批次/时期更新模型参数的多少。较小的值会导致学习速度较慢，而较大的值可能会导致训练过程中不可预测的行为。】

learning_rate = 1e-3
batch_size = 64
epochs = 5

3 Optimization Loop

Once we set our hyperparameters, we can then train and optimize our model with an optimization loop. Each iteration of the optimization loop is called an epoch.

【一旦我们设置了超参数，我们就可以通过优化循环训练和优化我们的模型。优化循环的每次迭代称为epoch。】

Each epoch consists of two main parts:

【每个epoch由两个主要部分组成:】

The Train Loop - iterate over the training dataset and try to converge to optimal parameters.
【训练循环-遍历训练数据集，并试图收敛到最优参数。】
The Validation/Test Loop - iterate over the test dataset to check if model performance is improving.
【验证/测试循环-对测试数据集进行迭代，以检查模型性能是否正在改善。】

Let’s briefly familiarize ourselves with some of the concepts used in the training loop. Jump ahead to see the Full Implementation of the optimization loop.

【让我们简单地熟悉一下训练循环中使用的一些概念。跳到前面，查看优化循环的完整实现。】

3.1 Loss Function

When presented with some training data, our untrained network is likely not to give the correct answer. Loss function measures the degree of dissimilarity of obtained result to the target value, and it is the loss function that we want to minimize during training. To calculate the loss we make a prediction using the inputs of our given data sample and compare it against the true data label value.

【当出现一些训练数据时，我们未经训练的网络很可能不会给出正确的答案。Loss function度量得到的结果与目标值的不相似程度，是我们在训练过程中希望最小化的Loss function。为了计算损失，我们使用给定数据样本的输入进行预测，并将其与真实的数据标签值进行比较。】

Common loss functions include nn.MSELoss (Mean Square Error) for regression tasks, and nn.NLLLoss (Negative Log Likelihood) for classification. nn.CrossEntropyLoss combines nn.LogSoftmax and nn.NLLLoss.

【常用的损失函数包括nn.MSELoss(Mean Square Error)用于回归；nn.NLLLoss(负对数似然)用于分类。nn.CrossEntropyLoss结合了nn.LogSoftmax 和 nn.NLLLoss。】

We pass our model’s output logits to nn.CrossEntropyLoss, which will normalize the logits and compute the prediction error.

【我们将模型的输出logit传递给nn.CrossEntropyLoss，将对数归一化并计算预测误差。】

# Initialize the loss function
loss_fn = nn.CrossEntropyLoss()

3.2 Optimizer

Optimization is the process of adjusting model parameters to reduce model error in each training step. Optimization algorithms define how this process is performed (in this example we use Stochastic Gradient Descent). All optimization logic is encapsulated in the optimizer object. Here, we use the SGD optimizer; additionally, there are many different optimizers available in PyTorch such as ADAM and RMSProp, that work better for different kinds of models and data.

【优化是在每个训练步骤中调整模型参数以减少模型误差的过程。优化算法定义了这个过程是如何执行的(在这个例子中，我们使用随机梯度下降法)。所有优化逻辑都封装在优化器对象中。这里，我们使用SGD优化器；此外，PyTorch中还有许多不同的优化器，比如ADAM和RMSProp，它们可以更好地处理不同类型的模型和数据。】

We initialize the optimizer by registering the model’s parameters that need to be trained, and passing in the learning rate hyperparameter.

【我们通过注册需要训练的模型参数，并传入学习速率超参数来初始化优化器。】

optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

Inside the training loop, optimization happens in three steps:

【在训练循环中，优化分为三个步骤:】

Call optimizer.zero_grad() to reset the gradients of model parameters. Gradients by default add up; to prevent double-counting, we explicitly zero them at each iteration.
【调用optimizer.zero_grad()重置模型参数的梯度。梯度默认情况下是叠加的；为了防止重复计算，我们在每次迭代时显式地将它们归零。】
Backpropagate the prediction loss with a call to loss.backwards(). PyTorch deposits the gradients of the loss w.r.t. each parameter.
【通过调用loss.backward()反向传播预测损失。PyTorch保存每个参数的损耗w.r.t.的梯度。】
Once we have our gradients, we call optimizer.step() to adjust the parameters by the gradients collected in the backward pass.
【一旦我们有了梯度，我们调用optimizer.step()来通过向后传递中收集的梯度来调整参数。】

4 Full Implementation

We define train_loop that loops over our optimization code, and test_loop that evaluates the model’s performance against our test data.

【我们定义了对优化代码进行循环的train_loop，以及针对测试数据评估模型性能的test_loop。】

def train_loop(dataloader, model, loss_fn, optimizer):
    size = len(dataloader.dataset)
    for batch, (X, y) in enumerate(dataloader):
        # Compute prediction and loss
        pred = model(X)
        loss = loss_fn(pred, y)

        # Backpropagation
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if batch % 100 == 0:
            loss, current = loss.item(), batch * len(X)
            print(f"loss: {loss:>7f}  [{current:>5d}/{size:>5d}]")


def test_loop(dataloader, model, loss_fn):
    size = len(dataloader.dataset)
    num_batches = len(dataloader)
    test_loss, correct = 0, 0

    with torch.no_grad():
        for X, y in dataloader:
            pred = model(X)
            test_loss += loss_fn(pred, y).item()
            correct += (pred.argmax(1) == y).type(torch.float).sum().item()

    test_loss /= num_batches
    correct /= size
    print(f"Test Error: \n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \n")

We initialize the loss function and optimizer, and pass it to train_loop and test_loop. Feel free to increase the number of epochs to track the model’s improving performance.

【我们初始化loss函数和优化器，并将其传递给train_loop和test_loop。您可以随意增加epoch的数量，以跟踪模型的改进性能。】

loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)

epochs = 10
for t in range(epochs):
    print(f"Epoch {t+1}\n-------------------------------")
    train_loop(train_dataloader, model, loss_fn, optimizer)
    test_loop(test_dataloader, model, loss_fn)
print("Done!")

输出结果：

Epoch 1
-------------------------------
loss: 2.290156  [    0/60000]
loss: 2.275099  [ 6400/60000]
loss: 2.256799  [12800/60000]
loss: 2.252760  [19200/60000]
loss: 2.235528  [25600/60000]
loss: 2.205756  [32000/60000]
loss: 2.204928  [38400/60000]
loss: 2.172354  [44800/60000]
loss: 2.160271  [51200/60000]
loss: 2.127511  [57600/60000]
Test Error:
 Accuracy: 49.9%, Avg loss: 2.116347

Epoch 2
-------------------------------
loss: 2.124757  [    0/60000]
loss: 2.107859  [ 6400/60000]
loss: 2.045332  [12800/60000]
loss: 2.061512  [19200/60000]
loss: 2.002954  [25600/60000]
loss: 1.940844  [32000/60000]
loss: 1.962774  [38400/60000]
loss: 1.874285  [44800/60000]
loss: 1.875532  [51200/60000]
loss: 1.802694  [57600/60000]
Test Error:
 Accuracy: 58.7%, Avg loss: 1.794751

Epoch 3
-------------------------------
loss: 1.830118  [    0/60000]
loss: 1.797928  [ 6400/60000]
loss: 1.670504  [12800/60000]
loss: 1.718298  [19200/60000]
loss: 1.605203  [25600/60000]
loss: 1.560042  [32000/60000]
loss: 1.583883  [38400/60000]
loss: 1.483568  [44800/60000]
loss: 1.515428  [51200/60000]
loss: 1.414553  [57600/60000]
Test Error:
 Accuracy: 62.0%, Avg loss: 1.430290

Epoch 4
-------------------------------
loss: 1.499763  [    0/60000]
loss: 1.472005  [ 6400/60000]
loss: 1.319050  [12800/60000]
loss: 1.399100  [19200/60000]
loss: 1.283040  [25600/60000]
loss: 1.279892  [32000/60000]
loss: 1.300507  [38400/60000]
loss: 1.221794  [44800/60000]
loss: 1.262865  [51200/60000]
loss: 1.173478  [57600/60000]
Test Error:
 Accuracy: 63.9%, Avg loss: 1.193923

Epoch 5
-------------------------------
loss: 1.268049  [    0/60000]
loss: 1.260393  [ 6400/60000]
loss: 1.092561  [12800/60000]
loss: 1.205449  [19200/60000]
loss: 1.083632  [25600/60000]
loss: 1.101792  [32000/60000]
loss: 1.134809  [38400/60000]
loss: 1.062815  [44800/60000]
loss: 1.108174  [51200/60000]
loss: 1.035161  [57600/60000]
Test Error:
 Accuracy: 65.1%, Avg loss: 1.049588

Epoch 6
-------------------------------
loss: 1.114492  [    0/60000]
loss: 1.130664  [ 6400/60000]
loss: 0.944653  [12800/60000]
loss: 1.083935  [19200/60000]
loss: 0.961972  [25600/60000]
loss: 0.981254  [32000/60000]
loss: 1.033072  [38400/60000]
loss: 0.961604  [44800/60000]
loss: 1.007507  [51200/60000]
loss: 0.948494  [57600/60000]
Test Error:
 Accuracy: 66.0%, Avg loss: 0.956025

Epoch 7
-------------------------------
loss: 1.006542  [    0/60000]
loss: 1.046684  [ 6400/60000]
loss: 0.842564  [12800/60000]
loss: 1.002121  [19200/60000]
loss: 0.884486  [25600/60000]
loss: 0.895794  [32000/60000]
loss: 0.965427  [38400/60000]
loss: 0.895181  [44800/60000]
loss: 0.937755  [51200/60000]
loss: 0.889426  [57600/60000]
Test Error:
 Accuracy: 67.3%, Avg loss: 0.891673

Epoch 8
-------------------------------
loss: 0.926312  [    0/60000]
loss: 0.987333  [ 6400/60000]
loss: 0.768049  [12800/60000]
loss: 0.943189  [19200/60000]
loss: 0.831892  [25600/60000]
loss: 0.833098  [32000/60000]
loss: 0.916814  [38400/60000]
loss: 0.850216  [44800/60000]
loss: 0.887719  [51200/60000]
loss: 0.846100  [57600/60000]
Test Error:
 Accuracy: 68.5%, Avg loss: 0.844885

Epoch 9
-------------------------------
loss: 0.864126  [    0/60000]
loss: 0.941802  [ 6400/60000]
loss: 0.711602  [12800/60000]
loss: 0.898299  [19200/60000]
loss: 0.793915  [25600/60000]
loss: 0.786041  [32000/60000]
loss: 0.879356  [38400/60000]
loss: 0.818412  [44800/60000]
loss: 0.850554  [51200/60000]
loss: 0.812724  [57600/60000]
Test Error:
 Accuracy: 69.7%, Avg loss: 0.809041

Epoch 10
-------------------------------
loss: 0.814177  [    0/60000]
loss: 0.904296  [ 6400/60000]
loss: 0.667563  [12800/60000]
loss: 0.862825  [19200/60000]
loss: 0.764706  [25600/60000]
loss: 0.750034  [32000/60000]
loss: 0.848550  [38400/60000]
loss: 0.794559  [44800/60000]
loss: 0.821466  [51200/60000]
loss: 0.785530  [57600/60000]
Test Error:
 Accuracy: 70.9%, Avg loss: 0.780144

Done!