pytorch 训练时raise EOFError EOFError

最新推荐文章于 2024-06-02 17:13:47 发布

flysnownet

最新推荐文章于 2024-06-02 17:13:47 发布

阅读量2.3k

点赞数

CC 4.0 BY-SA版权

分类专栏： python 文章标签： pytorch python 人工智能

本文链接：https://blog.youkuaiyun.com/flysnownet/article/details/133844643

python 专栏收录该内容

3 篇文章

订阅专栏

文章描述了在Cyclegan模型训练过程中，使用DataLoader获取验证数据时遇到的EOFError异常。通过设置torch的multiprocessing共享策略为file_system，作者解决了这个问题。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

训练到一半时获取验证数据报错

报错代码

imgs = next(iter(val_dataloader))

    val_dataloader = DataLoader(
        ImageDataset("data/%s" % opt.dataset_name, transforms_=transforms_, unaligned=True, mode="test"),
        batch_size=5,
        shuffle=True,
        num_workers=2,
    )

def sample_images(batches_done):
    """Saves a generated sample from the test set"""
    imgs = next(iter(val_dataloader))
    G_AB.eval()
    G_BA.eval()
    real_A = Variable(imgs["A"].type(Tensor))
    fake_B = G_AB(real_A)
    real_B = Variable(imgs["B"].type(Tensor))
    fake_A = G_BA(real_B)

错误

File "/content/cyclegan/cyclegan.py", line 324, in <module>
    sample_images(batches_done)
  File "/content/cyclegan/cyclegan.py", line 56, in sample_images
    imgs = next(iter(val_dataloader))
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 633, in __next__
    data = self._next_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1328, in _next_data
    idx, data = self._get_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1294, in _get_data
    success, data = self._try_get_data()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 1132, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.10/multiprocessing/queues.py", line 122, in get
    return _ForkingPickler.loads(res)
  File "/usr/local/lib/python3.10/dist-packages/torch/multiprocessing/reductions.py", line 307, in rebuild_storage_fd
    fd = df.detach()
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 58, in detach
    return reduction.recv_handle(conn)
Traceback (most recent call last):
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 145, in _serve
    send(conn, destination_pid)
  File "/usr/lib/python3.10/multiprocessing/resource_sharer.py", line 50, in send
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 189, in recv_handle
    reduction.send_handle(conn, new_fd, pid)
    return recvfds(s, 1)[0]
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 184, in send_handle
  File "/usr/lib/python3.10/multiprocessing/reduction.py", line 159, in recvfds
    raise EOFError
EOFError

解决方法

尝试了很多方法没解决，最终找到了解决方案
在dataset.py里加上以下代码

import torch.multiprocessing
torch.multiprocessing.set_sharing_strategy('file_system')