fused_adam.so: cannot open shared object file: No such file or directory

已于 2024-03-07 11:05:51 修改
阅读量5.1k 收藏 20
点赞数 30
文章标签： python
于 2024-03-07 10:45:42 首次发布
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/robator/article/details/136512416
版权
文章讲述了用户在使用deepspeed进行分布式训练时遇到的安装错误，主要原因是CUDA版本（nvcc11.2）与torch版本（2.0.1+cu117）不匹配。通过调整CUDA版本和尝试编译操作，最终发现torch的CUDA版本需与nvcc版本一致才能成功安装fused_adam等扩展。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

最近使用分布式训练框架deepspeed进行训练，安装后报错，如下所示

 File "**/site-packages/deepspeed/ops/adam/fused_adam.py", line 94, in __init__
    fused_adam_cuda = FusedAdamBuilder().load()
    op_module = load(name=self.name,  File "**/site-packages/deepspeed/ops/op_builder/builder.py", line 479, in load

  File "**/site-packages/torch/utils/cpp_extension.py", line 1284, in load
    return self.jit_load(verbose)
  File "**/site-packages/deepspeed/ops/op_builder/builder.py", line 523, in jit_load
    op_module = load(name=self.name,
    return _jit_compile(  File "**/site-packages/torch/utils/cpp_extension.py", line 1284, in load

  File "**/site-packages/torch/utils/cpp_extension.py", line 1535, in _jit_compile
    return _jit_compile(
  File "**/site-packages/torch/utils/cpp_extension.py", line 1535, in _jit_compile
    return _import_module_from_library(name, build_directory, is_python_module)
  File "**/site-packages/torch/utils/cpp_extension.py", line 1929, in _import_module_from_library
    return _import_module_from_library(name, build_directory, is_python_module)
  File "**/site-packages/torch/utils/cpp_extension.py", line 1929, in _import_module_from_library
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py38_cu117/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory
    module = importlib.util.module_from_spec(spec)
  File "<frozen importlib._bootstrap>", line 556, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1166, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /root/.cache/torch_extensions/py38_cu117/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory

检查版本如下，操作系统centos7,内核3.10.0-1160.92.1.el7.x86_64 ，python 3.8,显卡驱动对应的版本11.2，torch 版本2.0.1+cu117，nvcc 版本11.2，deepspeed 版本0.13.4
ds_resport 输出如下

[2024-03-06 17:33:52,658] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.0
 [WARNING]  using untested triton version (2.0.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['**/site-packages/torch']
torch version .................... 2.0.1+cu117
deepspeed install path ........... ['**/site-packages/deepspeed']
deepspeed info ................... 0.13.4, unknown, unknown
torch cuda version ............... 11.7
torch hip version ................ None
nvcc version ..................... 11.2
deepspeed wheel compiled w. ...... torch 1.10, cuda 10.2
shared memory (/dev/shm) size .... 125.87 GB
 

其中fused_adam ............. [NO] ....... [OKAY] 显示未安装，其中torch cuda 版本为11.7，但nvcc 版本为11.2；更换一台机器，ubuntu 20.04，python 3.10,显卡驱动对应的cuda 12.2nvcc 版本为11.8,torch 版本2.1.2+cu121，deepspeed 0.13.5

运行deepspeed 报错同上

运行ds_report

[2024-03-06 17:47:43,578] [INFO] [real_accelerator.py:191:get_accelerator] Setting ds_accelerator to cuda (auto detect)
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
      runtime if needed. Op compatibility means that your system
      meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
 [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
 [WARNING]  async_io: please install the libaio-dev package with apt
 [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
async_io ............... [NO] ....... [NO]
fused_adam ............. [NO] ....... [OKAY]
cpu_adam ............... [NO] ....... [OKAY]
cpu_adagrad ............ [NO] ....... [OKAY]
cpu_lion ............... [NO] ....... [OKAY]
 [WARNING]  Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
evoformer_attn ......... [NO] ....... [NO]
fused_lamb ............. [NO] ....... [OKAY]
fused_lion ............. [NO] ....... [OKAY]
inference_core_ops ..... [NO] ....... [OKAY]
cutlass_ops ............ [NO] ....... [OKAY]
transformer_inference .. [NO] ....... [OKAY]
quantizer .............. [NO] ....... [OKAY]
ragged_device_ops ...... [NO] ....... [OKAY]
ragged_ops ............. [NO] ....... [OKAY]
random_ltd ............. [NO] ....... [OKAY]
 [WARNING]  sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.1
 [WARNING]  using untested triton version (2.1.0), only 1.0.0 is known to be compatible
sparse_attn ............ [NO] ....... [NO]
spatial_inference ...... [NO] ....... [OKAY]
transformer ............ [NO] ....... [OKAY]
stochastic_transformer . [NO] ....... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['**/site-packages/torch']
torch version .................... 2.1.2+cu121
deepspeed install path ........... ['**/site-packages/deepspeed']
deepspeed info ................... 0.13.5, unknown, unknown
torch cuda version ............... 12.1
torch hip version ................ None
nvcc version ..................... 11.8
deepspeed wheel compiled w. ...... torch 2.1, cuda 12.1
shared memory (/dev/shm) size .... 125.76 GB其中torch cuda 版本12.1，但nvcc 版本为11.8，此时按照参考资料2的内容，复制github上deepspeed目录，进入deepspeed目录，并执行DS_BUILD_FUSED_ADAM=1 pip3 install .，报如下错误：

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "**/DeepSpeed/setup.py", line 196, in <module>
          ext_modules.append(builder.builder())
        File "**/DeepSpeed/op_builder/builder.py", line 633, in builder
          assert_no_cuda_mismatch(self.name)
        File "**/DeepSpeed/oop_builder/builder.py", line 101, in assert_no_cuda_mismatch
          raise CUDAMismatchException(
      op_builder.builder.CUDAMismatchException: >- DeepSpeed Op Builder: Installed CUDA version 11.8 does not match the version torch was compiled with 12.1, unable to compile cuda/cpp extensions without a matching cuda version.
      DS_BUILD_OPS=0
       [WARNING]  async_io requires the dev libaio .so object and headers but these were not found.
       [WARNING]  async_io: please install the libaio-dev package with apt
       [WARNING]  If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
显示cuda 版本11.8和12.1版本的torch 不匹配，因此升级cuda 为12.2，然后再进入deepspeed目录，DS_BUILD_FUSED_ADAM=1 pip3 install . ，安装后ds_report，发现fused_adam已安装上，fused_adam ............. [YES] ...... [OKAY]，再次执行训练，发现已经不在报上述错误，注意DS_BUILD_FUSED_ADAM=1 pip3 install deepspeed 是不起作用的，安装不上fused_adam

由此得出结论：torch 的cuda 版本要和nvcc 的版本一致才可以，至少torch 的cuda 版本不能比nvcc 的版本低才行（也不能太高）

参考资料：

1 fused_adam.so: cannot open shared object file: No such file or directory问题排查与解决-优快云博客2 fused_adam.so: cannot open shared object file: No such file or directory · Issue #119 · databrickslabs/dolly · GitHub

3  【工程实践】解决 nvcc: command not found_nvcc -v 提示未找到命令-优快云博客

4   https://github.com/stanford-crfm/mistral/issues/196

robator
关注
deepspeed使用zero3 + offload报错:AttributeError: ‘DeepSpeedCPUAdam‘ object has no attribute ‘ds_opt_adam
qq_44193969的博客
03-26 1万+
说是系统cuda和torch的cuda版本不匹配，我们直接改成不检查cuda版本。3、修改完后再次在命令行执行。输出以下内容表示大功告成。
Ubuntu中报错：No module named ‘fused‘ / ‘upfirdn2d‘ 问题分析及详细解决方案
weixin_46115959的博客
03-28 1435
出现报错。
fused_adam/fused_adam.so: cannot open shared object file: No such file
qq_44013369的博客
03-12 1923
可能的原因是transformers版本过高。
Linux 快速部署DeepSeek-R1 蒸馏系列模型
MnivL的博客
02-08 3015
本文记录使用 ms-Swift 框架 对 DeepSeekR1 蒸馏系列模型（7B、14B）在 A100（40GB）* 1 服务器上部署过程与测试结果。
fused_adam.so: cannot open shared object file: No such file or directory问题排查与解决
qq_35284646的博客
07-14 5838
fused_adam.so: cannot open shared object file: No such file or directory问题排查与解决
编译自定义算子出现的问题（GCFSR）-- fused.so: No such file or directory
在人工智能的道路上爬行
07-20 1097
使用conda重新创建一个环境进行编译，编译成功。在进行编译，且编译成功，
deepspeed训练模型提示：cpu_adam.so: cannot open shared object file: No such file or directory
weixin_39663060的博客
07-31 1170
我们只需要在安装deepspeed时设置DS_BUILD_CPU_ADAM=1即可。因此，我们卸载并重新安装deepspeed。，可以自己alias一下或者建立符号链接到PATH目录中。然后就会自动编译出cpu_adam.so文件。至此，本人问题解决。
cannot open shared object file: No such file or directory解决方法
weixin_41038905的博客
07-05 2万+
对自己打包的so文件进行测试，编译不报错 但是运行时 出现错误： cannot open shared object file: No such file or directory 但是so文件明明已经放在当前目录下了,折腾一下午，发现这是linux下so库经常会出现的错误，测试程序的可执行文件名字为start， 用ldd start可以发现程序寻找每个so的路径基本为系统路径，lib或者u...
fused.so: cannot open shared object file: No such file or directory
07-25
- *1* [fused_adam.so: cannot open shared object file: No such file or directory问题排查与解决](https://blog.csdn.net/qq_35284646/article/details/125785970)[target="_blank" data-report-click={"spm":...
ImportError: /home/lin/.cache/torch_extensions/py310_cu118/fused/fused.so: cannot open shared object file: No such file or directory
12-28
当遇到 `ImportError: cannot open shared object file` 的错误提示时，通常意味着 Python 试图加载的共享库文件不存在于指定路径下。对于特定情况下的 `fused.so` 文件缺失问题，在 PyTorch 扩展模块中可能出现此...
安装mmdet3d报错【fatal error: spconv/maxpool.h: No such file or directory】
最新发布
Garfield2005的专栏
02-19 637
但是想确认谁提供的这个头文件，还是比较麻烦的，需要研究源码的实现。所以笔者采取偷懒的方法：先看源码中是否有该文件（包括源码中依赖的第三方源码），方法也和简单：直接整个目录下搜索该文件。从上图可以看到，目录下有maxpool.h、mp_helper.h、spconv_ops.h等各种头文件，正式之前编译时报错的这些文件。那就奇怪了，源码中有需要的头文件，编译命令中也包含了该include的路径，怎么会找不到呢？至此，报错原因找到，源码打包上传python库时有问题，导致pip安装时下载的源码不对。
【已解决】使用TorchDrug时，ninja编译错误或找不到torch_ext
xuran
05-09 1247
问题1： c++: error: unrecognized command line option ‘-std=c++14’ ninja: build stopped: subcommand failed. 问题2： ImportError: No module named ‘torch_ext’ 我是首先出现第一个问题，采用下边第③种方案，然后提示问题2，没有解决，最后在看警告的时候，发现提醒GCC版本问题，所以就将服务器上的4.8升级到5.5，最终在服务器上执行py代码是ok的。另外，如果本地的Pych
cannot open shared object file: No such file or directory
n_fly的博客
01-01 1330
一、linux下调用动态库.so文件时提示： cannot open shared object file: No such file or directory 解决办法： 1、此时ldd xxx查看依赖缺少哪些库 libmysqlcppconn.so.7 => not found libboost_system.so.1.64.0 => not found 2、设置变量 LD_LIBRARY_PATH vim ~/.bashrc 最后一行加上： export L...
Linux动态库常见问题之-“cannot open shared object file No such file or directory“的解决办法
热门推荐
飞翔de刺猬
02-25 11万+
Linux编写程序时，会经常的接触动态库，而在程序运行时可能会遇到类似于:libxxx.so cannot open shared object file No such file or directory的问题，这个代表什么含义，该怎么解决呢？ 含义 这个问题代表的含义是，可执行程序在加载libxxx.so库时，找不到该库。至于找不到的情况分为两种:系统里根本不存在libxxx.so库；libx...
Xtuner
qq_43650129的博客
08-01 1730
在完成 XTuner 的安装后，我们下一步就需要去明确我们自己的微调目标了。对于全量微调的模型（full）其实是不需要进行整合这一步的，因为全量微调修改的是原模型的权重而非微调一个新的 Adapter ，因此是不需要进行模型整合的。在 PART 1 的部分，由于我们不再需要在 HuggingFace 上自动下载模型，因此我们先要更换模型的路径以及数据集的路径为我们本地的路径。，在使用前我们需要准备好三个路径，包括原模型的路径、训练好的 Adapter 层的（模型格式转换后的）路径以及最终保存的路径。
部署大模型时的一些bug
weixin_45925624的博客
11-05 1457
2）跑demo，可以pip uninstall flash_attn。PyTorch 和 cuda 版本不匹配。更改下的模型的 config.json。1）重装pytorch 1.13.1。更改 —output。
apex报错error: subprocess-exited-with-error
weixin_50898138的博客
10-17 1042
apex报错error: subprocess-exited-with-error
【软件安装】安装deepspeed 时 cuda 报错
Damu
09-06 6546
这种情况一般是由于cuda 重新安装或没有在默认路径。找到自己的cuda 安装路径后，使。下，因此没有找到系统环境变量。
评论 1
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值