RTX3090 tensorflow1.x报错:Blas GEMM launch failed

之前一直使用的3090跑pytorch代码,无事发生。但跑开源tensorlfow1.x代码的时候出现了问题,主要有两种情况:

  1. 有的代码报错:Blas GEMM launch failed
  2. 有的代码能正常运行,但加载十分缓慢,在跑起来之前大概能加载5分钟

因为使用tf.test.is_gpu_available()检查返回结果一直是True,一时让我搞不清楚遇到了什么问题。后来排查发现,3090不支持cuda10,cuda9等。虽然使用anaconda创建的环境安装tensorflow1.x的时候自动选择了cuda,但cuda和3090是不适配的。

目前解决方法主要有三种:

  1. 从源码自己编译tensorflow(因为没尝试过,就简要提一下)
  2. 利用官方针对A100编译的版本,Accelerating TensorFlow on NVIDIA A100 GPUs
  3. 使用官方提供的docker镜像(docker

通过方法2我解决了问题,
主要运行以下两个命令

pip install nvidia-pyindex
pip install nvidia-tensorflow

参考的方法:
自己源码编译
利用官方编译的版本
docker版本安装

2025-03-11 10:44:54.222546: E T:\src\github\tensorflow\tensorflow\stream_executor\cuda\cuda_blas.cc:654] failed to run cuBLAS routine cublasSgemm_v2: CUBLAS_STATUS_EXECUTION_FAILED Traceback (most recent call last): File "C:\Users\19124\anaconda3\envs\rl(tf1.x)\lib\site-packages\tensorflow\python\client\session.py", line 1322, in _do_call return fn(*args) File "C:\Users\19124\anaconda3\envs\rl(tf1.x)\lib\site-packages\tensorflow\python\client\session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "C:\Users\19124\anaconda3\envs\rl(tf1.x)\lib\site-packages\tensorflow\python\client\session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(128, 7), b.shape=(7, 128), m=128, n=128, k=7 [[Node: Critic/dense/MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_state_0_0/_5, Critic/dense/kernel/read)]] [[Node: Critic/dense_1/BiasAdd/_7 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_21_Critic/dense_1/BiasAdd", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "D:\threeMotorsProject\threeMotorsProject\RL\PPO\ppo.py", line 157, in <module> train() File "D:\threeMotorsProject\threeMotorsProject\RL\PPO\ppo.py", line 122, in train ppo.update(np.vstack(buffer_s), np.vstack(buffer_a), np.array(discounted_r)[:, np.newaxis]) File "D:\threeMotorsProject\threeMotorsProject\RL\PPO\ppo.py", line 80, in update adv = self.sess.run(self.v, {self.S: s}) - r File "C:\Users\19124\anaconda3\envs\rl(tf1.x)\lib\site-packages\tensorflow\python\client\session.py",
最新发布
03-12
评论 12
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值