[torch]Getting the Output of a Layer

最新推荐文章于 2023-07-12 11:30:58 发布
最新推荐文章于 2023-07-12 11:30:58 发布
阅读量467 收藏
点赞数
CC 4.0 BY-SA版权
分类专栏： torch
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/apsvvfb/article/details/71565839
本文通过实例演示了如何使用Torch库构建一个包含FastLSTM层的序列模型，并详细展示了模型内部各层输出的状态，有助于理解序列模型的工作原理。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

https://groups.google.com/forum/#!topic/torch7/R9DAdx95aqc

introduction

require 'cutorch'
require 'cunn'
require 'rnn'
require 'os'

tensor1 = torch.zeros(5,10)
tensor1[3]=torch.rand(1,10)
print(tensor1)
input = {tensor1,torch.rand(5,10),torch.rand(5,10),torch.rand(5,10),torch.rand(5,10),torch.rand(5,10)}
net = nn.Sequencer(
   nn.Sequential()
      :add(nn.MaskZero(nn.FastLSTM(10,3),1))
      :add(nn.MaskZero(nn.Linear(3,4),1))
      :add(nn.MaskZero(nn.LogSoftMax(),1))
)

output = net:forward(input)
local m = net.modules
--[[
print("net")
print(net)
print("m")
print(m)
--]]
for i = 1, #input do
        print(output[i])
        print(m[1].sharedClones[i].modules[1].output)
end

print("net")
print(net)
print("m")
print(m)

output

net 
nn.Sequencer @ nn.Recursor @ nn.Sequential {
  [input -> (1) -> (2) -> (3) -> output]
  (1): nn.MaskZero @ nn.FastLSTM(10 -> 3)
  (2): nn.MaskZero @ nn.Linear(3 -> 4)
  (3): nn.MaskZero @ nn.LogSoftMax
}
m   
{
  1 : 
    {
      sharedClones : 
        {
          1 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
          2 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
          3 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
          4 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
          5 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
          6 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
        }
      step : 7
      outputs : 
        {
          1 : DoubleTensor - size: 5x4
          2 : DoubleTensor - size: 5x4
          3 : DoubleTensor - size: 5x4
          4 : DoubleTensor - size: 5x4
          5 : DoubleTensor - size: 5x4
          6 : DoubleTensor - size: 5x4
        }
      output : DoubleTensor - size: 5x4
      gradInput : DoubleTensor - empty
      modules : 
        {
          1 : 
            {
              gradInput : DoubleTensor - empty
              modules : 
                {
                  1 : {...}
                  2 : {...}
                  3 : {...}
                }
              _type : "torch.DoubleTensor"
              output : DoubleTensor - size: 5x4
            }
        }
      _gradOutputs : {...}
      rho : 6
      recurrentModule : 
        {
          gradInput : DoubleTensor - empty
          modules : 
            {
              1 : 
                {
                  output : DoubleTensor - size: 5x3
                  gradInput : DoubleTensor - empty
                  nInputDim : 1
                  batchmode : true
                  zeroMask : ByteTensor - size: 5x1
                  _type : "torch.DoubleTensor"
                  _zeroMask : DoubleTensor - size: 5x1
                  module : {...}
                  modules : {...}
                }
              2 : 
                {
                  output : DoubleTensor - size: 5x4
                  gradInput : DoubleTensor - empty
                  nInputDim : 1
                  batchmode : true
                  zeroMask : ByteTensor - size: 5x1
                  _type : "torch.DoubleTensor"
                  _zeroMask : DoubleTensor - size: 5x1
                  module : {...}
                  modules : {...}
                }
              3 : 
                {
                  output : DoubleTensor - size: 5x4
                  gradInput : DoubleTensor - empty
                  nInputDim : 1
                  batchmode : true
                  zeroMask : ByteTensor - size: 5x1
                  _type : "torch.DoubleTensor"
                  _zeroMask : DoubleTensor - size: 5x1
                  module : {...}
                  modules : {...}
                }
            }
          _type : "torch.DoubleTensor"
          output : DoubleTensor - size: 5x4
        }
      nSharedClone : 6
      _type : "torch.DoubleTensor"
      gradInputs : {...}
      module : 
        {
          gradInput : DoubleTensor - empty
          modules : 
            {
              1 : 
                {
                  output : DoubleTensor - size: 5x3
                  gradInput : DoubleTensor - empty
                  nInputDim : 1
                  batchmode : true
                  zeroMask : ByteTensor - size: 5x1
                  _type : "torch.DoubleTensor"
                  _zeroMask : DoubleTensor - size: 5x1
                  module : {...}
                  modules : {...}
                }
              2 : 
                {
                  output : DoubleTensor - size: 5x4
                  gradInput : DoubleTensor - empty
                  nInputDim : 1
                  batchmode : true
                  zeroMask : ByteTensor - size: 5x1
                  _type : "torch.DoubleTensor"
                  _zeroMask : DoubleTensor - size: 5x1
                  module : {...}
                  modules : {...}
                }
              3 : 
                {
                  output : DoubleTensor - size: 5x4
                  gradInput : DoubleTensor - empty
                  nInputDim : 1
                  batchmode : true
                  zeroMask : ByteTensor - size: 5x1
                  _type : "torch.DoubleTensor"
                  _zeroMask : DoubleTensor - size: 5x1
                  module : {...}
                  modules : {...}
                }
            }
          _type : "torch.DoubleTensor"
          output : DoubleTensor - size: 5x4
        }
      rmInSharedClones : true
    }
}

可以看出m是一个table类型的变量. 所以看看想要它输出什么就能输出什么.
例如:

tensor1 = torch.zeros(5,10)
tensor1[3]=torch.rand(1,10)
print(tensor1)
input = {tensor1,torch.rand(5,10),torch.rand(5,10),torch.rand(5,10),torch.rand(5,10),torch.rand(5,10)}
net = nn.Sequencer(
   nn.Sequential()
      :add(nn.MaskZero(nn.FastLSTM(10,3),1))
--      :add(nn.MaskZero(nn.Linear(3,4),1))
--      :add(nn.MaskZero(nn.LogSoftMax(),1))
)

output = net:forward(input)
local m = net.modules
--[[
print("net")
print(net)
print("m")
print(m)
--]]
for i = 1, #input do
        print(output[i])
        print(m[1].sharedClones[i].modules[1].output)
end)

test

require 'cutorch'
require 'cunn'
require 'rnn'
require 'os'
--[[
net = nn.Sequencer(
   nn.Sequential()
      :add(nn.MaskZero(nn.FastLSTM(10,6),1))
      :add(nn.MaskZero(nn.Linear(6,4),1))
      :add(nn.MaskZero(nn.LogSoftMax(),1))
)
parameters, gradParameters = net:getParameters()
lightModel = net:clone('weight','bias','running_mean','running_std')
torch.save('model.t7',lightModel)
--]]

net=torch.load("model.t7")

--[[
tensor1 = torch.zeros(5,10)
tensor1[3]=torch.Tensor{3,4,5,6,7,8,23,2,12,90}
tensor2 = torch.ones(5,10)
tensor2[{{1,2},{}}]=torch.Tensor{ {1,3,4,5,6,0,3,2,56,2}, {5,3,2,5,7,3,45,78,235,10}}
tensor2[4]=torch.ones(1,10):fill(3.2)
tensor2[5]=torch.zeros(1,10)
input = {tensor1,tensor2}
--]]
--net=torch.load("/work1/t2g-shinoda2011/15M54105/trecvid/torch-lstm3/batch5_epoch5_hiddensize256_cw1/model_100ex_batch5_unit256_epoch70")
--[[
array = {}
tensor1  = torch.zeros(5,10)
tensor1[3]=torch.rand(1,10)
tensor2 = torch.rand(5,10)
tensor3 = torch.rand(5,10)
tensor4 = torch.rand(5,10)
tensor1=tensor1:cuda()
tensor2=tensor2:cuda()
tensor3=tensor3:cuda()
tensor4=tensor4:cuda()
table.insert(array, tensor1)
table.insert(array, tensor2)
table.insert(array, tensor3)
table.insert(array, tensor4)
file = torch.DiskFile('input.asc', 'w')
file:writeObject(array)
file:close()
os.exit()
--]]
net:cuda()
file = torch.DiskFile('input.asc', 'r')
input = file:readObject()
print(input)
local m = net.modules
output = net:forward(input)
--[[
print("net")
print(net)
print("m")
print(m)
--]]
model = (nn.MaskZero(nn.LogSoftMax(),1)):cuda()
for seqj = 1, #input do
    print(seqj)
    res = m[1].sharedClones[seqj].modules[2].output
    out1=output[seqj]
    out2=model:forward(res)
    print(out1-out2)

end

test.lua得到的tep = m[1].sharedClones[seqj].modules[2].output[i]是取了net的第二层的输出.
out1与out2值相等.说明确实tep是第二层的输出.

《Python基础教程》内容总览篇（持续更新中）
weixin_43178406的博客
08-26 30万+
大家好，我是爱编程的喵喵。双985硕士毕业，现担任全栈工程师一职，热衷于将数据思维应用到工作与生活中。从事机器学习以及相关的前后端开发工作。曾在阿里云、科大讯飞、CCF等比赛获得多次Top名次。现为优快云博客专家、人工智能领域优质创作者。喜欢通过博客创作的方式对所学的知识进行总结与归纳，不仅形成深入且独到的理解，而且能够帮助新手快速入门。个人精心开设的《Python基础课程》专栏订阅量接近900，帮助不少同学解决了Bug。
【trition-server】pytorch 文档：使用 Triton 提供 Torch-TensorRT 模型
突围
07-05 412
Serving a Torch-TensorRT model with Triton
Pytorch中的张量数据类型（Tensor）
hanmo22357的博客
03-14 5944
数学中有标量、向量和矩阵的概念，它们的维度分别是0、1、2。也就是说，标量中元素的位置固定，向量中元素的位置需要通过其索引确定，矩阵中的元素位置需要通过其行号和列号确定。张量可以视为矩阵的扩展，可以用于表示无穷维度的数据。张量（Tensor）是Pytorch库中的基本数据类型，Pytorch中各种基本数字类型都有其对应的Tensor类型，但是在Pytorch中没有内嵌的字符串类型。
DoubleTensor
weixin_42280069的博客
12-09 733
[外链图片转存失败,源站可能有防盗在这里插入!链机制,建描述]议将图片上https://传(imblog.csq223img.cn/20131209125171.png32989)(一上午的错误https://img-blog.csdnimg.cn/20191209132925171.png)] 精度不匹配，肯定是代码中即出现了doule 又出现了float，统一一下， 可以这样修改，也可以都改为...
Torch学习笔记
whgyxy的博客
08-14 1万+
Torch笔记 （二）快速入门 Torch中的唯一的数据结构就是Tensor了，而该结构简洁而且强大，非常适合进行矩阵类的数值计算，它是Torch中最最重要的类了。这个Tensor其实就是个多维矩阵，支持矩阵的各种操作。这里需要特别强调的是，lua中的数组（其实是table）下标是从1开始的，因此Tensor对象的下标也是从1开始的。 咱们从程序猿的角度来说，首先，Tensor也是有类型的
PyTorch中Tensor的数据类型和运算
ch_musk的博客
06-28 2万+
在使用Tensor时，我们首先要掌握如何使用Tensor来定义不同数据类型的变量。Tensor时张量的英文，表示多维矩阵，和numpy对应，PyTorch中的Tensor可以和numpy的ndarray相互转换，唯一不同的是PyTorch可以在GPU上运行，而numpy的ndarray只能在cpu上运行。 常用的不同数据类型的Tensor，有32位的浮点型torch.F...
pytorch中Tensor的数据类型
热门推荐
moshiyaofei的博客
04-30 4万+
1. 32位浮点型：torch.FloatTensor a=torch.Tensor( [[2,3],[4,8],[7,9]], ) print "a:",a print "a.size():",a.size() print "a.dtype:",a.dtype b=torch.FloatTensor( [[2,3],[4,8],[7,9]] ) print "b:",b print "b....
Parameters for big model inference torch_dtype (str or torch.dtype, optional) — Override the default torch.dtype and load the model under a specific dtype. The different options are: torch.float16 or torch.bfloat16 or torch.float: load in a specified dtype, ignoring the model’s config.torch_dtype if one exists. If not specified the model will get loaded in torch.float (fp32). "auto" - A torch_dtype entry in the config.json file of the model will be attempted to be used. If this entry isn’t found then next check the dtype of the first weight in the checkpoint that’s of a floating point type and use that as dtype. This will load the model using the dtype it was saved in at the end of the training. It can’t be used as an indicator of how the model was trained. Since it could be trained in one of half precision dtypes, but saved in fp32. A string that is a valid torch.dtype. E.g. “float32” loads the model in torch.float32, “float16” loads in torch.float16 etc. For some models the dtype they were trained in is unknown - you may try to check the model’s paper or reach out to the authors and ask them to add this information to the model’s card and to insert the torch_dtype entry in config.json on the hub. device_map (str or dict[str, Union[int, str, torch.device]] or int or torch.device, optional) — A map that specifies where each submodule should go. It doesn’t need to be refined to each parameter/buffer name, once a given module name is inside, every submodule of it will be sent to the same device. If we only pass the device (e.g., "cpu", "cuda:1", "mps", or a GPU ordinal rank like 1) on which the model will be allocated, the device map will map the entire model to this device. Passing device_map = 0 means put the whole model on GPU 0. To have Accelerate compute the most optimized device_map automatically, set device_map="auto". For more information about each option see designing a device map. max_memory (Dict, optional) — A dictionary device identifier to maximum memory if using device_map. Will default to the maximum memory available for each GPU and the available CPU RAM if unset. tp_plan (str, optional) — A torch tensor parallel plan, see here. Currently, it only accepts tp_plan="auto" to use predefined plan based on the model. Note that if you use it, you should launch your script accordingly with torchrun [args] script.py. This will be much faster than using a device_map, but has limitations. tp_size (str, optional) — A torch tensor parallel degree. If not provided would default to world size. device_mesh (torch.distributed.DeviceMesh, optional) — A torch device mesh. If not provided would default to world size. Used only for tensor parallel for now. offload_folder (str or os.PathLike, optional) — If the device_map contains any value "disk", the folder where we will offload weights. offload_state_dict (bool, optional) — If True, will temporarily offload the CPU state dict to the hard drive to avoid getting out of CPU RAM if the weight of the CPU state dict + the biggest shard of the checkpoint does not fit. Defaults to True when there is some disk offload. offload_buffers (bool, optional) — Whether or not to offload the buffers with the model parameters. quantization_config (Union[QuantizationConfigMixin,Dict], optional) — A dictionary of configuration parameters or a QuantizationConfigMixin object for quantization (e.g bitsandbytes, gptq). There may be other quantization-related kwargs, including load_in_4bit and load_in_8bit, which are parsed by QuantizationConfigParser. Supported only for bitsandbytes quantizations and not preferred. consider inserting all such arguments into quantization_config instead. subfolder (str, optional, defaults to "") — In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. variant (str, optional) — If specified load weights from variant filename, e.g. pytorch_model..bin. variant is ignored when using from_tf or from_flax. use_safetensors (bool, optional, defaults to None) — Whether or not to use safetensors checkpoints. Defaults to None. If not specified and safetensors is not installed, it will be set to False. weights_only (bool, optional, defaults to True) — Indicates whether unpickler should be restricted to loading only tensors, primitive types, dictionaries and any types added via torch.serialization.add_safe_globals(). When set to False, we can load wrapper tensor subclass weights. key_mapping (`dict[str, str], optional) — A potential mapping of the weight names if using a model on the Hub which is compatible to a Transformers architecture, but was not converted accordingly. kwargs (remaining dictionary of keyword arguments, optional) — Can be used to update the configuration object (after it being loaded) and initiate the model (e.g., output_attentions=True). Behaves differently depending on whether a config is provided or automatically loaded: If a configuration is provided with config, **kwargs will be directly passed to the underlying model’s __init__ method (we assume all relevant updates to the configuration have already been done) If a configuration is not provided, kwargs will be first passed to the configuration class initialization function (from_pretrained()). Each key of kwargs that corresponds to a configuration attribute will be used to override said attribute with the supplied kwargs value. Remaining keys that do not correspond to any configuration attribute will be passed to the underlying model’s __init__ function. 详细解释一下上面这个
最新发布
07-25
- 详细映射字典：如`{"layer1": 0, "layer2": "cpu"}` - **特殊值**： - `0` 或 `"cuda:0"`：整个模型放在GPU 0 - `"mps"`：Apple Silicon芯片专用 - **示例**： ```python # 自动分配模型到可用设备 model = ...
torch.nn、(二)
weixin_36670529的博客
08-30 3146
目录 Recurrent layers RNN LSTM GRU RNNCell LSTMCell GRUCell Transformer layers Transformer TransformerEncoder TransformerDecoder TransformerEncoderLayer TransformerDecoderLayer Linear laye...
torch分布式训练笔记
魏圣颖的博客
07-12 391
这里假设模型只有四个参数，分成两个 bucket，反向传播时，bucket0中的两个参数的梯度信息完成更新后，两个进程就可以分别调用 allreduce 操作，完成梯度信息的通信和同步。最基础的想法是，所有的卡都完成了整个模型的反向传播和梯度更新后，开始通信。但实际上每张卡可能效率不同，有的卡完成了全部梯度的更新，有的还没有。其他：关于 forward 和 backwark 中更多细节，比如通信具体是如何触发的， hook 的实现，这些后续阅读代码细节时再做介绍。DDP的构造函数中把上面说的流程都做了。
Python 矩阵之上12 OnMatrix2-tensor和numpy的其他基础操作
yukai08008的博客
12-17 888
说明 内容
解决报错Input type (torch.cuda.DoubleTensor) and weight type (torch.cuda.FloatTensor) should be the same
qq_34612816的博客
03-09 2万+
解决RuntimeError: Input type and weight type should be the same 今天用pytorch编写代码时，遇到了一个错误（如下图所示）。 根据报错信息的意思可以推断，这个错误是由输入和权重的数据类型不一致引起的。 更多关于torch中张量（tensor）数据类型的介绍，可参考这个网页Link。 因此解决方法很简单，就是将输入的数据和模型参数的数据类型统一即可。在这个例子中，有以下几个解决方法。 1.将输入数据（torch.tensor 形式）转换成Floa
Tensor类型之间的相互转换
wangshuhuan1的博客
07-29 1万+
遇到了Tensor类型之前需要相互转换的问题，就记录一下，顺便分享。
Torch初学（一）
sunflower_Yolanda的博客
06-21 1万+
Tensor多维矩阵，可以使用LongStorage --- creation of a 4D-tensor 4x5x6x2 z = torch.Tensor(4,5,6,2) --- for more dimensions, (here a 6D tensor) one can do: s = torch.LongStorage(6) s[1] = 4; s[2] = 5; s[3] =
Torch教程(1)-Tensor
南有乔木NTU的博客
01-04 1万+
Torch是一个广泛支持机器学习算法的科学计算框架。易于使用且高效，主要得益于一个简单的和快速的脚本语言LuaJIT，和底层的C / CUDA实现。 核心特征的总结： 1. 一个强大的n维数组 2. 很多实现索引，切片，移调transposing的例程 3. 惊人的通过LuaJIT的C接口 4. 线性代数例程 5. 神经网络，并基于能量的模型 6. 数值优化例程 7. 快速高效的GP
【PyTorch】Tensor和tensor的区别
玄云飘风的博客
12-29 3万+
本文列举的框架源码基于PyTorch1.0，交互语句在0.4.1上测试通过 import torch 在PyTorch中，Tensor和tensor都能用于生成新的张量： >>> a=torch.Tensor([1,2]) >>> a tensor([1., 2.]) >>> a=torch.tensor([1,2]) >>&gt...
expected type torch.cuda.DoubleTensor but got torch.cuda.FloatTensor python报错
是鲤鱼啊
05-06 6152
错误提示需要变量是DoubleTensor类型，但得到的却是FloatTensor类型。于是我把报错的那一行涉及到的两个变量都转为了DoubleTensor类型，果然就好了，见下图： ...
【解决方案】Expected object of type torch.FloatTensor but found type torch.DoubleTensor
gc.collect()
11-14 6343
简述 在网上找了半天没有得到解答。 但是，在pycharm输入.之后，发现有一个函数很有意思，然后得到了解决 解决方案 假如报错的语句是 model是模型 input是输入(tensor的实体) model(input) 方法其实很简单 mode(input.float()) ...
深度学习框架Torch7解析-- Tensor篇
余昌黔|书山有路
05-24 1万+
Tensor 类型 多维矩阵 数据的内部表示 内存管理 常用函数 Tensor 的构造函数 常用操作函数 复制和初始化 Rsizing 提取子tensor 未完待续 TensorTensor类可以说是Torch中最重要的类了,因为几乎所有的包都依赖于这个类实现.它是整个Torch实现的数据基础.类型Tensor由以下几种类型:ByteTensor -- 代表 unsigned char C
please explain the code of torch.optim.AdamW
06-10
Certainly. Here is an example code snippet for using torch.optim.AdamW: ``` import torch import torch.optim as optim # Define the model model = torch.nn.Linear(10, 1) # Define the optimizer optimizer = optim.AdamW(model.parameters(), lr=0.001, weight_decay=0.01) # Define the loss function criterion = torch.nn.MSELoss() # Train the model for epoch in range(100): optimizer.zero_grad() outputs = model(inputs) loss = criterion(outputs, targets) loss.backward() optimizer.step() ``` In this code snippet, we first define a simple linear model with 10 input features and 1 output feature. We then define an instance of the AdamW optimizer by passing in the model's parameters, a learning rate of 0.001, and a weight decay of 0.01. We also define a mean squared error loss function (MSELoss) to use for training the model. Finally, we train the model for 100 epochs using a loop. In each epoch, we zero out the gradients, compute the outputs of the model, compute the loss, backpropagate the loss to compute the gradients, and update the model's parameters using the optimizer's step method. This is a simple example of how to use torch.optim.AdamW for training a neural network.
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值