【pytorch DistributedDataParallel 及amp 使用过程遇到的问题记录】

Magicapprentice

已于 2024-10-20 10:23:02 修改

阅读量1.6k

点赞数 30

文章标签： pytorch 人工智能 python

于 2024-10-20 09:18:03 首次发布

本文链接：https://blog.youkuaiyun.com/Magicapprentice/article/details/142998770

版权

环境

pytorch==2.3.1

问题

单机多卡时：

超时错误

部分报错内容:

Heartbeat monitor timed out! Process will be terminated after dumping debug info. workMetaList_.size()=2
[rank1]:[E ProcessGroupNCCL.cpp:1153] [PG 0 Rank 1] ProcessGroupNCCL preparing to dump debug info.
[rank1]:[F ProcessGroupNCCL.cpp:1169] [PG 0 Rank 1] [PG 0 Rank 1] ProcessGroupNCCL’s watchdog got stuck for 600 seconds without making progress in monitoring enqueued collectives

报错截图:
在这里插入图片描述

解决方法:

代码中:


dist.init_process_group(backend='nccl', init_method='env://',rank=self.opt.local_rank, world_size=self.opt.world_size,)

改为:

dist.init_process_group(backend='nccl', init_method='env://',rank=self.opt.local_rank, world_size=self.opt.world_size,  timeout=timedelta(seconds=3600),)

存在没有使用梯度的参数

（不使用ddp时正常，使用ddp时报错）

报错内容:

If you already have done the above, then the distributed data parallel module wasn’t able to locate the output tensors in the return value of your module’s forward function. Please include the loss function and the structure of the return value of forward of your module when reporting this issue (e.g. list, dict, iterable).
[rank0]: Parameter indices which did not receive grad for rank 0: 60 6