DeepLizard学习笔记

DeepLizard：油管地址、对应博客笔记

1. 创建tensor

创建tensor的四种方法即不同：具体解释

2. Tensor Operation Types

• Reshaping operations
• Element-wise operations
• Reduction operations
• Access operations

2.1 Reshaping operations

1. 常见reshape的操作 链接
   
   
2. CNN最后一层的flatten操作（）
   1）构建
   
   2）Flatten()
   这里我们希望保留第一层：指定start_dim参数
   

2.2 Element-wise operations

原文链接
Broadcasting And Element-Wise Operations

逐元素计算要求不同tensor具有相同shape，broadcasting解决了tensor间不同shape的问题。

Broadcasting is the concept whose implementation allows us to add scalars to higher dimensional tensors.

Comparison Operations Are Element-Wise

Element-Wise Operations Using Functions

2.3 Tensor Reduction Operations

原文链接
A reduction operation on a tensor is an operation that reduces the number of elements contained within the tensor.

Reduction operations in general allow us to compute aggregate (total) values across data structures. In our case, our structures are tensors.

Reducing Tensors By Axes
注意图中右边的解释

Argmax Tensor Reduction Operation
Argmax returns the index location of the maximum value inside a tensor.

2.4 Accessing Elements Inside Tensors

原文链接

With NumPy ndarray objects, we have a pretty robust set of operations for indexing and slicing, and PyTorch tensor objects support most of these operations as well.

2.5 总结

3. CNN Examples

There are four general steps that we’ll be following as we move through this project:

• Prepare the data
• Build the model
• Train the model
• Analyze the model’s results

3.1 CNN Image Preparation

原文

To prepare our data, we’ll be following an ETL process：

• Extract data from a data source.
• Transform data into a desirable format.
• Load data into a suitable structure.

Once we have completed the ETL process, we are ready to begin building and training our deep learning model.

Preparing Our Data Using PyTorch

Our ultimate goal when preparing our data is to do the following (ETL):

• Extract – Get the Fashion-MNIST image data from the source.
• Transform – Put our data into tensor form.
• Load – Put our data into an object to make it easily accessible.

For these purposes, PyTorch provides us with two classes:


但是，这里使用torchvision包，其中对fashion-MNIST dataset的这些操作已经进行了封装，但是其背后的原理就是上面的Dataset和DataLoader.

PyTorch Torchvision Package

The torchvision package, gives us access to the following resources:

• Datasets (like MNIST and Fashion-MNIST)
• Models (like VGG16)
• Transforms
• Utils

3.2 DataSet和DataLoader

原文
PyTorch Dataset: Working With The Training Set
PyTorch DataLoader: Working With Batches Of Data

3.3 Build the model

3.3.1 理解nn.module

原文
从python类的角度考虑pytorch中的Module类。

PyTorch’s nn.Module Class

下面是对nn.Module很好的理解：

deep neural networks are built using multiple layers. This is what makes the network deep.
Each layer in a neural network has two primary components:

• A transformation (code)
• A collection of weights (data)

Like many things in life, this fact makes layers great candidates to be represented as objects using OOP(object oriented programming)

Within the nn package, there is a class called Module, and it is the base class for all of neural network modules which includes layers.

This means that all of the layers in PyTorch extend the nn.Module class and inherit all of PyTorch’s built-in functionality within the nn.Module class. In OOP this concept is known as inheritance.

Even neural networks extend the nn.Module class. This makes sense because neural networks themselves can be thought of as one big layer (if needed, let that sink in over time).

forward() Method：

Each layer has its own transformation (code) and the tensor passes forward through each layer. The composition of all the individual layer forward passes defines the overall forward pass transformation for the network.

The goal of the overall transformation is to transform or map the input to the correct prediction output class, and during the training process, the layer weights (data) are updated in such a way that cause the mapping to adjust to make the output closer to the correct prediction.

What this all means is that, every PyTorch nn.Module has a forward() method, and so when we are building layers and networks, we must provide an implementation of the forward() method. The forward method is the actual transformation.

PyTorch’s nn.functional Package:

When we implement the forward() method of our nn.Module subclass, we will typically use functions from the nn.functional package. This package provides us with many neural network operations that we can use for building layers. In fact, many of the nn.Module layer classes use nn.functional functions to perform their operations.

The nn.functional package contains methods that subclasses of nn.Module use for implementing their forward() functions. Later, we see an example of this by looking at the PyTorch source code of the nn.Conv2d convolutional layer class.

Building A Neural Network In PyTorch:

Short version:

• Extend the nn.Module base class.
• Define layers as class attributes.
• Implement the forward() method.

More detailed version:

• Create a neural network class that extends the nn.Module base class.
• In the class constructor, define the network’s layers as class attributes using pre-built layers from torch.nn.
• Use the network’s layer attributes as well as operations from the nn.functional API to define the network’s forward pass.

3.3.2 CNN Layers

原文
Two Types Of Parameters：

• Hyperparameters：values are chosen manually and arbitrarily
  例：
  

• Data dependent hyperparameters：whose values are dependent on data

CNN Layers参数意义：

3.3.3 CNN Weights - Learnable Parameters In Neural Networks

原文
Hyperparameter values are chosen arbitrarily.
learnable parameters are the weights inside our network, and they live inside each layer.

In fact, when we say that a network is learning, we specifically mean that the network is learning the appropriate values for the learnable parameters. Appropriate values are values that minimize the loss function.

PyTorch Parameter Class：
To keep track of all the weight tensors inside the network, PyTorch has a special class called Parameter. The Parameter class extends the tensor class, and so the weight tensor inside every layer is an instance of this Parameter class.

3.3.4 Callable Neural Networks

原文

How Linear Layers Work

Callable Layers And Neural Networks

> fc(in_features)
tensor([30.0261, 40.1404, 49.7643], grad_fn=<AddBackward0>)

Instead of calling the forward() method directly, we call the object instance. After the object instance is called, the __ call__() method is invoked under the hood, and the __ call__() in turn invokes the forward() method. This applies to all PyTorch neural network modules, namely, networks and layers.

any time we want to invoke our forward() method, we call the object instance.

3.3.5 CNN Forward Pass Implementation

原文

一些非常好的解释：

Each of these layers is comprised of a collection of weights (data) and a collection operations (code). The weights are encapsulated inside the nn.Conv2d() class instance. The relu() and the max_pool2d() calls are just pure operations. Neither of these have weights, and this is why we call them directly from the nn.functional API.

Sometimes we may see pooling operations referred to as pooling layers. Sometimes we may even hear activation operations called activation layers.
However, what makes a layer distinct from an operation is that layers have weights. Since pooling operations and activation functions do not have weights, we will refer to them as operations and view them as being added to the collection of layer operations.
For example, we’ll say that the second layer in our network is a convolutional layer that contains a collection of weights, and preforms three operations, a convolution operation, the relu activation operation, and the max pooling operation.

Note that the rules and terminology here are not strict. This is just one way to describe a network. There are other ways to express these ideas. The main thing we need to be aware of is which operations are defined using weights and which ones don’t use any weights.

Historically, the operations that are defined using weights are what we call layers. Later, other operations were added to the mix like activation functions and pooling operations, and this caused some confusion in terminology.

Mathematically, the entire network is just a composition of functions, and a composition of functions is a function itself. So a network is just a function. All the terms like layers, activation functions, and weights, are just used to help describe the different parts.

Don’t let these terms confuse the fact that the whole network is simply a composition of functions, and what we are doing now is defining this composition inside our forward() method.

完整的forward方法：

3.3.6 Forward Propagation Explained

原文

1. 关闭计算图自动计算梯度
   在单纯观察前向传播时，不需要pytorch自动计算梯度。
   Before we being, we are going to turn off PyTorch’s gradient calculation feature. This will stop PyTorch from automatically building a computation graph as our tensor flows through the network.

> torch.set_grad_enabled(False) 
<torch.autograd.grad_mode.set_grad_enabled at 0x17c4867dcc0>

2. CNN输入与结果预测
   需要输入CNN的数据格式是：(batch_size, in_channels, height, width)
   如果是单张图像，没有batch_size，通过unsqueeze填补：
   

3. batch Processing
   

4. CNN Output Size Formula
   

3.3.7 CNN Training

原文

CNN training loop:

3.3.8 CNN Confusion Matrix

原文
This confusion matrix will allow us to see which categories our network is confusing with one another.
这个视频解决了当时困惑很久的不使用grad的问题，视频中对于梯度的解释和confusion matrix的构建更加详细具体，建议观看整个视频。

Locally Disabling PyTorch Gradient Tracking

1. 使用装饰器@torch.no_grad()
   @torch.no_grad()：omit gradient tracking.
   
   This is because gradient tracking uses memory, and during inference (getting predictions while not training) there is no need to keep track of the computational graph. The decoration is one way of locally turning off the gradient tracking feature while executing specific functions.

2. with torch.no_grad():

with torch.no_grad():
    prediction_loader = torch.utils.data.DataLoader(train_set, batch_size=10000)
    train_preds = get_all_preds(network, prediction_loader)

Building The Confusion Matrix

1. 自己写
2. sklearn.metrics中的confusion_matrix()

3.3.9 Concatenate Vs Stack

原文

• Concatenating joins a sequence of tensors along an existing axis
• Stacking joins a sequence of tensors along a new axis.

1. Add Or Insert An Axis Into A Tensor：

2. Stack Vs Cat In PyTorch：


3. 实例分析：
1）Joining Images Into A Single Batch：

2）Joining Batches Into A Single Batch：

3）Joining Images With An Existing Batch

3.4 Analyze the model’s results

3.4.1 TensorBoard: TensorFlow’s Visualization Toolkit

原文

后面30-36暂且跳过。

4. 其他的补充

4.1 PyTorch Sequential Models

原文
The Sequential class make it much easier to rapidly build networks and allows us to skip over the step where we implement the forward() method. When we use the sequential way of building a PyTorch network, we construct the forward() method implicitly by defining our network’s architecture sequentially.

This means that we can

• compose layers to make networks
• since networks are also nn.Module instances, we can also compose networks with one another.
• since the Sequential class is also a nn.Module itself, we can even compose Sequential modules with one another.

1.Building PyTorch Sequential Networks

2.Class Definition Vs Sequential

We said that these networks are the same. But what do we mean? In this case, we mean that the networks have the same architecture. From a programming standpoint, the two networks are different types under the hood.