reform513的博客

梯度是怎么计算的import torch a = torch.randn((3,3), requires_grad = True)w1 = torch.randn((3,3), requires_grad = True)w2 = torch.randn((3,3), requires_grad = True)w3 = torch.randn((3,3), requires_grad = True)w4 = torch.randn((3,3), requires_grad = Tru...

1. 动态图 vs 静态图pytorch 是动态建图，好处是方便使用python control flow statements控制计算流走向，每个iteration可以有完全不同的计算图，同时也比较容易处理shape不确定的tensors or variables。除此，dynamic graphs are debug friendly, becuase it allows for line by line execution of code and you can have access to al

torch.distributed支持3中backends，'gloo', 'mpi' 和 'nccl', 每种backend支持的函数是不同的，需要查表确认https://pytorch.org/docs/stable/distributed.html#下面以backend = 'nccl' 为例， 演示获取分布在不同GPU上的某个tensor的值的方式1. reduce or all_reduce # 获取该tensor的均值box_iou = comput...