多级多卡分布式训练时，报错 RuntimeError: Socket Timeout

最新推荐文章于 2024-07-09 20:19:52 发布

明泽.

最新推荐文章于 2024-07-09 20:19:52 发布

阅读量5.3k

点赞数

文章标签：分布式 pytorch 深度学习

本文链接：https://blog.youkuaiyun.com/qq_36955294/article/details/123113283

版权

在尝试使用Python的torch库进行多进程分布式训练时遇到了问题。具体表现为在初始化过程组时发生了SocketTimeout错误，这可能与网络连接超时或者集群内部通信故障有关。报错发生在torch的multiprocessing和distributed模块中。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

Traceback (most recent call last):
  File "distribute_prune_erfnet_cluster.py", line 643, in <module>
    daemon=False, )
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 200, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 158, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 119, in join
    raise Exception(msg)
Exception: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/spawn.py", line 20, in _wrap
    fn(i, *args)
  File "/workspace/geniii-trainingcode-lane/lanenet/distribute_prune_erfnet_cluster.py", line 601, in _distributed_worker
    distributed.init_process_group(backend="gloo", init_method=dist_url, world_size=world_size, rank=global_rank)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 432, in init_process_group
    timeout=timeout)
  File "/opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 503, in _new_process_group_helper
    timeout=timeout)
RuntimeError: Socket Timeout