解决RuntimeError: CUDA error: an illegal memory access was encountered问题

最新推荐文章于 2025-11-10 10:06:17 发布

原创最新推荐文章于 2025-11-10 10:06:17 发布 · 1.1w 阅读

3 ·

CC 4.0 BY-SA版权

文章标签：

#pytorch #深度学习 #python #人工智能 #nlp

小bug 专栏收录该内容

2 篇文章

订阅专栏

在使用PyTorch进行深度学习，特别是多GPU训练transformers模型时，遇到RuntimeError: CUDA error: an illegal memory access was encountered。经过排查，发现不是tensor未分配到GPU导致，而是由于batchsize过大，内存占用过多引起的。减小batchsize可以避免该错误。

部署运行你感兴趣的模型镜像

用pytorch在多卡训练transformers的时候出现了以下问题：

RuntimeError: CUDA error: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f7b645511e2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7f7b6479ff92 in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f7b6453f9cd in /usr/local/lib/python3.6/dist-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x540ae2 (0x7f7bb00d7ae2 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x540b86 (0x7f7bb00d7b86 in /usr/local/lib/python3.6/dist-packages/torch/lib/libtorch_python.so)
frame #5: /usr/bin/python() [0x554316]
frame #6: /usr/bin/python() [0x585d2c]
frame #7: /usr/bin/python() [0x585d2c]
frame #8: /usr/bin/python() [0x573087]
frame #9: /usr/bin/python() [0x5542f2]
frame #10: /usr/bin/python() [0x586108]
frame #11: /usr/bin/python() [0x5aa838]
frame #12: /usr/bin/python() [0x5aa84e]
frame #13: /usr/bin/python() [0x5aa84e]
frame #14: /usr/bin/python() [0x56e336]
frame #15: PyDict_SetItemString + 0x156 (0x575256 in /usr/bin/python)
frame #16: PyImport_Cleanup + 0x76 (0x4ed566 in /usr/bin/python)
frame #17: Py_FinalizeEx + 0x5e (0x637efe in /usr/bin/python)
frame #18: Py_Main + 0x37e (0x638f5e in /usr/bin/python)
frame #19: main + 0xe0 (0x4a6f10 in /usr/bin/python)
frame #20: __libc_start_main + 0xe7 (0x7f7bb3412b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: _start + 0x2a (0x5afa0a in /usr/bin/python)