复制+粘贴 -＞ 集群就绪 | SLURM HPC

SLURM HPC集群快速搭建指南
原创 已于 2022-08-28 15:23:26 修改 · 934 阅读 · 1 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#centos #ubuntu #机器学习 #自动化

于 2022-04-18 00:15:50 首次发布
本文提供了一种适合非专业Linux运维人员的SLURM HPC集群搭建方案，无需复杂技能，只需简单的复制粘贴指令。适用于CentOS和Ubuntu系统，通过hpc4you toolkit工具，实现集群组建流程的零配置、零管理和零维护，特别适合科研团队快速建立计算平台。

请看这里. 

http://t.csdn.cn/KXzn5icon-default.png?t=M7J4http://t.csdn.cn/KXzn5 

写在前面

本方案, 是假定您的计算机技能, 停留在多数简历里面描述的“熟练掌握Micosoft Word, PowerPoint, Excel; 会使用Origin绘图, 会PhotoShop简单修图”这个水平. 但是您可以区分清楚键盘上的字母键, 数字键, 方向键, 以及粘贴完毕指令后, 还要按Enter这种操作逻辑. 

本方案半自动调试集群. 
  • 如果能用vi, 仅需用vi修改一个文件. (不要求会用, 会用vi得多牛呀. 能用vi添加几行内容和会用vi是两回事情)
最低0.47元/天 解锁文章
hpc4you
关注
复制+粘贴-＞集群就绪 | 微软视角hpc4you_toolkit使用演示
hpc4you的博客
05-12 662
此文写给生物、化学、环境、材料以及其他需要计算模拟的硕士和博士一年级新生. hpc4you_toolkit可以轻松快捷地利用现有机器组建并行计算集群. 无论你的服务器是运行CentOS还是Ubuntu, hpc4you_toolkit组建集群, 都是依次运行如下指令: ./step1.sh ./step2.sh ./step3.sh ./step4.sh...............
ACK集群上实现Slurm HPC & Kubernetes负载混合调度
2401_88127808的博客
12-14 1324
为什么要提出在ACK集群上实现Slurm HPC & Kubernetes负载混合调度呢？原因分析：目前ACK会提供静态分配+分离调度的方案，但是由于每个Slurm Pod的规格固定，且Slurm Pod属于提前占用集群资源，Slurm集群中资源空闲时Kubernetes无法使用这些已占用的集群资源，从而导致集群资源碎片。此外，修改Slurm Pod的资源规格需要删除Pod重建，因此在Slurm与Kubernetes资源占用变化较大的场景中，节点迁移的难度较大。
slurm-web, Slurm HPC工作负载管理器 web JS仪表板和 JSON REST API.zip
09-18
slurm-web, Slurm HPC工作负载管理器 web JS仪表板和 JSON REST API slurm站点 概述Slurm web提供了一个web仪表板和一个 REST API 到Slurm高性能超级计算机，具有当前作业和。后端 REST API 是使用 Flask web框架在 python 中完成的。 它被设计
slurm:适用于HPC计算模型专家的Docker容器中的入门级Slurm并行计算
03-07
slurm:适用于HPC计算模型专家的Docker容器中的入门级Slurm并行计算
slurmr-workshop：带有Slurm，R和slurmR软件包的HPC研讨会
02-19
关于研讨会 摘要Slurm作业计划程序（目前由USC使用）为您的所有计算需求提供了灵活的基础架构。 从提交单核长时间运行的任务到复杂的多节点任务，Slurm涵盖了计算资源管理方面所有科学家的需求。 R编程语言尽管不支持HPC，但具有多个基于社区的解决方案，可将您的计算管道与HPC设置（包括Slurm）集成在一起。 在本研讨会中，我们将说明如何将R与USC的HPC群集一起使用，涵盖从最基本的内容（例如提交简单的R脚本）到更复杂的功能设置，例如在节点中使用数十个或数百个内核的多节点套接字群集。相同的R会话。 该研讨会将由凯克医学院的南加州大学生物统计学部门主持。 议长 网站可以在查看 快乐科学家精选：Lynn Conway 林恩·康威（Lynn Conway），2006年7月。资料来源： Lynn Ann Conway（生于1938年1月2日）是美国计算机科学家，电气工程师，发明家和变性人
基于HPC场景的集群管理系统（slurm系统初相识）
刘炫320的博客
11-09 3896
LSF是一个比较主流的任务调度系统，包括深圳和苏州的超算中心都是使用的LSF系统，包括Spectrum LSF, PlatformLSF和OpenLava。我接触到的都是PlatformLSF。SGE则包括UGE和SGE，本来是开源的，但是被Oracle收购后就变成了闭源了。PBS则包括OpenPBS,PBS PRO 和Moab/TORQUE，可能由于比较老了吧，目前我没有见到了。
HPC调度基础：slurm集群的部署
天翼云开发者社区
03-30 2050
slurm是一种开源的、具有容错性和高度可扩展的Linux集群超级计算系统资源管理和作业调度系统。具体来说，计算集群使用slurm对资源和作业进行管理和调度，减少计算任务间相互干扰，提高集群运行效率。2016年世界十大超算系统，其中就有五个是使用slurm来进行管理的，甚至有众所周知的2016年冠军超算——神威太湖之光。slurm给超算系统的提供了三个关键功能：1.为用户分配一定时间的专享或非专享的资源(计算机节点)，以执行作业（Job）2.提供了用于启动、执行、监测在节点上运行的作业的框架。
slurm-web:Slurm HPC工作负载管理器Web JS仪表板和JSON REST API
04-30
Slurm-web为Slurm HPC超级计算机提供Web仪表板和REST API，并具有当前作业和节点状态的视图。 后端REST API是使用Flask Web框架在Python中开发的。 它旨在作为WSGI应用程序在HTTP服务器（例如Apache2）上运行。 它...
HPC (GPU)集群常用命令 - slurm作业调度方法 - ssh登录集群独占GPU(VS Code) - .sh/.py脚本的运行（持续更新）
科研汪小陈的主页：https://chensuzeyu.github.io
05-07 1782
【常用linux命令】HPC 集群常用命令 (封闭式管理的slurm调度系统) :​ 最近因加入了导师团队，需要将在AutoDL上的小规模实验数据+代码迁移到学校的HPC集群中进行大规模实验; 但现实很骨感，学校的集群是已封闭式管理的； 而且slurm工作的运行调度逻辑和AutoDL这类服务器有很大差别； 上传数据的单个文件大小卡在6GB以下，得分卷（避坑指南可参考博客）； 之后会持续更新一些个人经验 ​
HPC计算集群Slurm使用
LeviLizhi的博客
09-23 1590
srun [options] program命令属于交互式提交作业，有屏幕输出，但容易受网络波动影响，断网或关闭窗口会导致作业中断。如果不关心节点和时间限制，可简写为srun -N 1 -n 4 hostname。使用 srun 命令，配合 -p 分区名 参数，以及 -N 节点数 -n 核心数 进行提交。mixed：混合，节点在运行作业，但有些空闲 CPU 核，可接受新作业。-w cn[01-02] 指定提交作业到cn01,cn02节点。-x cn[01-02] 排除cn01、cn02节点。
使用 Slurm Federation 调度实现HPC混合云 Cloudburst
亚马逊云科技专栏
06-02 1300
背景Slurm集群调度系统支持创建集群联合（Federation），并在集群之间以对等方式调度作业。提交到联合集群的作业将收到唯一的作业ID，该ID在联合集群中的所有群集中都是唯一的。作业...
HPC&AI并行计算集群Slurm作业调度系统快速上手
qq_27815483的博客
09-19 1675
Slurm 是一个开源、容错、高度可扩展的集群管理和作业调度系统，适用于大型和小型 Linux 集群。Slurm 的操作不需要修改内核，而且相对独立。作为集群工作负载管理器，slurm 有三个关键功能。首先，它在一段时间内为用户分配对资源(计算节点)的独占和/或非独占访问权，以便他们能够执行工作。其次，它提供了一个框架，用于在分配的节点集上启动、执行和监视工作(通常是一个并行作业)。最后，它通过管理挂起的工作队列来仲裁资源争用。
centos7三节点部署slurm集群详细教程
wu852278685的博客
11-30 3594
​ 文章摘自本人笔记《【slurm】一、centos7三节点部署slurm集群详细教程》，地址：【slurm】一、centos7三节点部署slurm集群详细教程 - 空山人语-IT技术分享学习网站 更多教程可以关注空山人语-IT技术分享学习网站 ​
ubuntu SLURM搭建
qq_41867980的博客
02-22 5485
安装munge（每一个节点都需要安装） 这里有两台机器在一个局域网中，在/etc/hosts上分别配置好IP 和 主机名 IP Master IP Slave1 配置好后看看能不能互相ping通 ssh是否能连上，如果没有安装ssh运行 sudo apt install ssh 在/etc/ssh/sshd_config里的permitrootlogin设置为yes sudo su apt upgrade apt install munge 通过apt安装会找到stable版本，相比于手动
HPC集群之——NIS+NFS实现多节点之间/home数据共享+节点免密
weixin_41674971的博客
08-03 3109
HPC集群之——NIS+NFS实现多节点之间/home数据共享+节点免密
一键自动配置SLURM调度器
hpc4you的博客
05-26 965
简介 hpc4you toolkit solo, 适用于在Linux单机/工作站, 一键自动配置SLURM调度器 支持RHEL7和RHEL8及其兼容系统(比如CentOS 7.x, 8.x, Rocky Linux 8.x等) 支持Ubuntu 20.04和22.04系列及其兼容衍生版本. 如何使用 将文件code和压缩包hpc4you_toolkit-solo*.tgz放在目标服务器上 在以上两个文件所在路径, 打开终端, 输入指令 source code 耐心等待, 调度器自动就绪. ..
HPC&AI并行计算集群Slurm作业调度系统对通用资源（GRES）的调度
qq_27815483的博客
08-27 3324
Slurm 支持定义和调度任意通用 RESources 的功能 （GRES）。为特定 GRES 类型启用了其他内置功能， 包括图形处理单元 （GPU）、CUDA 多进程服务 （MPS） 设备，并通过可扩展的插件机制进行分片。
拥抱高性能计算：将Slurm集成为K8s调度程序
h_m_l的博客
11-27 7511
SUNK是一个开源项目（将于 2024 年初发布），它将 Kubernetes 容器化部署和 GitOps 引入 Slurm，并将 Slurm 调度程序插件集成到 Kubernetes。本质上，SUNK 将 Slurm 集成为 Kubernetes 调度程序，并允许 Slurm 作业在 Kubernetes 内运行。这创造了更加无缝的体验，在同一中央平台上支持爆发式和批量工作负载，并允许开发人员利用 Kubernetes 上的 SLURM 资源管理。
高性能计算的基石 - 简单易用的集群资源管理及作业调度系统
qq_21394345的博客
01-04 1444
Slurm是一个专为各种大小的Linux集群设计的开源。
Using nodes: slurm-gb200-217-[027,047] # nThread 1 nGpus 1 minBytes 536870912 maxBytes 8589934592 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # # Using devices # Rank 0 Group 0 Pid 234738 on slurm-gb200-217-027 device 0 [0x01] NVIDIA GB200 # Rank 1 Group 0 Pid 234739 on slurm-gb200-217-027 device 1 [0x01] NVIDIA GB200 # Rank 2 Group 0 Pid 234740 on slurm-gb200-217-027 device 2 [0x01] NVIDIA GB200 # Rank 3 Group 0 Pid 234741 on slurm-gb200-217-027 device 3 [0x01] NVIDIA GB200 # Rank 4 Group 0 Pid 237289 on slurm-gb200-217-047 device 0 [0x01] NVIDIA GB200 # Rank 5 Group 0 Pid 237290 on slurm-gb200-217-047 device 1 [0x01] NVIDIA GB200 # Rank 6 Group 0 Pid 237291 on slurm-gb200-217-047 device 2 [0x01] NVIDIA GB200 # Rank 7 Group 0 Pid 237292 on slurm-gb200-217-047 device 3 [0x01] NVIDIA GB200 slurm-gb200-217-027:234738:234738 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234738:234738 [0] NCCL INFO Bootstrap: Using eth0:10.0.4.227<0> slurm-gb200-217-027:234738:234738 [0] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-027:234738:234738 [0] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-027:234740:234740 [2] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-027:234740:234740 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234740:234740 [2] NCCL INFO Bootstrap: Using eth0:10.0.4.227<0> slurm-gb200-217-027:234740:234740 [2] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-027:234741:234741 [3] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-027:234741:234741 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234741:234741 [3] NCCL INFO Bootstrap: Using eth0:10.0.4.227<0> slurm-gb200-217-027:234741:234741 [3] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-027:234739:234739 [1] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-027:234739:234739 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234739:234739 [1] NCCL INFO Bootstrap: Using eth0:10.0.4.227<0> slurm-gb200-217-027:234739:234739 [1] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-047:237289:237289 [0] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-047:237289:237289 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237289:237289 [0] NCCL INFO Bootstrap: Using eth0:10.0.5.220<0> slurm-gb200-217-047:237289:237289 [0] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-047:237292:237292 [3] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-047:237292:237292 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237292:237292 [3] NCCL INFO Bootstrap: Using eth0:10.0.5.220<0> slurm-gb200-217-047:237292:237292 [3] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-047:237290:237290 [1] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-047:237290:237290 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237290:237290 [1] NCCL INFO Bootstrap: Using eth0:10.0.5.220<0> slurm-gb200-217-047:237290:237290 [1] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-027:234738:235129 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-027:234738:235129 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-027:234738:235129 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-027:234738:235129 [0] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-027:234738:235129 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-027:234738:235129 [0] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-027:234738:235129 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237291:237291 [2] NCCL INFO cudaDriverVersion 12080 slurm-gb200-217-047:237291:237291 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237291:237291 [2] NCCL INFO Bootstrap: Using eth0:10.0.5.220<0> slurm-gb200-217-047:237291:237291 [2] NCCL INFO NCCL version 2.25.1+cuda12.8 slurm-gb200-217-027:234738:235129 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-027:234738:235129 [0] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.4.227<0> slurm-gb200-217-027:234738:235129 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-027:234738:235129 [0] NCCL INFO Using network IBext_v8 slurm-gb200-217-027:234740:235130 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-027:234740:235130 [2] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-027:234741:235131 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-027:234741:235131 [3] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-027:234740:235130 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-027:234740:235130 [2] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-027:234741:235131 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-027:234741:235131 [3] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-027:234740:235130 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-027:234740:235130 [2] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-027:234741:235131 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-027:234741:235131 [3] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-027:234740:235130 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234741:235131 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234739:235132 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-027:234739:235132 [1] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-027:234739:235132 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-027:234739:235132 [1] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-027:234739:235132 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-027:234739:235132 [1] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-027:234739:235132 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234741:235131 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-027:234741:235131 [3] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.4.227<0> slurm-gb200-217-027:234740:235130 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-027:234740:235130 [2] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.4.227<0> slurm-gb200-217-027:234741:235131 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-027:234741:235131 [3] NCCL INFO Using network IBext_v8 slurm-gb200-217-027:234740:235130 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-027:234740:235130 [2] NCCL INFO Using network IBext_v8 slurm-gb200-217-027:234739:235132 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-027:234739:235132 [1] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.4.227<0> slurm-gb200-217-027:234739:235132 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-027:234739:235132 [1] NCCL INFO Using network IBext_v8 slurm-gb200-217-047:237289:237675 [0] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-047:237289:237675 [0] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-047:237289:237675 [0] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-047:237289:237675 [0] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-047:237289:237675 [0] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-047:237289:237675 [0] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-047:237289:237675 [0] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237292:237676 [3] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-047:237292:237676 [3] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-047:237292:237676 [3] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-047:237292:237676 [3] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-047:237292:237676 [3] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-047:237292:237676 [3] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-047:237292:237676 [3] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237290:237677 [1] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-047:237290:237677 [1] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-047:237290:237677 [1] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-047:237290:237677 [1] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-047:237290:237677 [1] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-047:237290:237677 [1] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-047:237290:237677 [1] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-027:234738:235129 [0] NCCL INFO DMA-BUF is available on GPU device 0 slurm-gb200-217-027:234738:235129 [0] NCCL INFO ncclCommInitRank comm 0xc4b2133f8d20 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 801000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-047:237289:237675 [0] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-047:237289:237675 [0] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.5.220<0> slurm-gb200-217-047:237292:237676 [3] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-047:237292:237676 [3] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.5.220<0> slurm-gb200-217-047:237290:237677 [1] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-047:237290:237677 [1] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.5.220<0> slurm-gb200-217-047:237289:237675 [0] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-047:237289:237675 [0] NCCL INFO Using network IBext_v8 slurm-gb200-217-047:237292:237676 [3] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-047:237292:237676 [3] NCCL INFO Using network IBext_v8 slurm-gb200-217-047:237290:237677 [1] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-047:237290:237677 [1] NCCL INFO Using network IBext_v8 slurm-gb200-217-047:237291:237678 [2] NCCL INFO NET/Plugin: Failed to find ncclNetPlugin_v9 symbol. slurm-gb200-217-047:237291:237678 [2] NCCL INFO NET/Plugin: Loaded net plugin NCCL RDMA Plugin v8 (v8) slurm-gb200-217-047:237291:237678 [2] NCCL INFO NET/Plugin: Failed to find ncclCollNetPlugin_v9 symbol. slurm-gb200-217-047:237291:237678 [2] NCCL INFO NET/Plugin: Loaded collnet plugin SHARP (v8) slurm-gb200-217-047:237291:237678 [2] NCCL INFO Plugin Path : /opt/hpcx/nccl_rdma_sharp_plugin/lib/libnccl-net.so slurm-gb200-217-047:237291:237678 [2] NCCL INFO P2P plugin v8 IBext_v8 slurm-gb200-217-047:237291:237678 [2] NCCL INFO NCCL_SOCKET_IFNAME set by environment to eth0 slurm-gb200-217-047:237291:237678 [2] NCCL INFO NCCL_IB_PCI_RELAXED_ORDERING set by environment to 1. slurm-gb200-217-047:237291:237678 [2] NCCL INFO NET/IB : Using [0]ibp0:1/IB/SHARP [1]ibp1:1/IB/SHARP [2]ibp2:1/IB/SHARP [3]ibp3:1/IB/SHARP [RO]; OOB eth0:10.0.5.220<0> slurm-gb200-217-047:237291:237678 [2] NCCL INFO PROFILER/Plugin: Could not find: libnccl-profiler.so. slurm-gb200-217-047:237291:237678 [2] NCCL INFO Using network IBext_v8 slurm-gb200-217-027:234741:235131 [3] NCCL INFO DMA-BUF is available on GPU device 3 slurm-gb200-217-027:234740:235130 [2] NCCL INFO DMA-BUF is available on GPU device 2 slurm-gb200-217-027:234741:235131 [3] NCCL INFO ncclCommInitRank comm 0xb4ab52367d50 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 1901000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-027:234739:235132 [1] NCCL INFO DMA-BUF is available on GPU device 1 slurm-gb200-217-027:234740:235130 [2] NCCL INFO ncclCommInitRank comm 0xabb25a0df930 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 1801000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-027:234739:235132 [1] NCCL INFO ncclCommInitRank comm 0xbf66c0b18250 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 901000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-027:234739:235132 [1] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-027:234740:235130 [2] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237290:237677 [1] NCCL INFO DMA-BUF is available on GPU device 1 slurm-gb200-217-047:237289:237675 [0] NCCL INFO DMA-BUF is available on GPU device 0 slurm-gb200-217-047:237290:237677 [1] NCCL INFO ncclCommInitRank comm 0xb82cf5a83890 rank 5 nranks 8 cudaDev 1 nvmlDev 1 busId 901000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-047:237289:237675 [0] NCCL INFO ncclCommInitRank comm 0xbdcbb5305fc0 rank 4 nranks 8 cudaDev 0 nvmlDev 0 busId 801000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-027:234741:235131 [3] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237289:237675 [0] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237292:237676 [3] NCCL INFO DMA-BUF is available on GPU device 3 slurm-gb200-217-047:237291:237678 [2] NCCL INFO DMA-BUF is available on GPU device 2 slurm-gb200-217-047:237292:237676 [3] NCCL INFO ncclCommInitRank comm 0xbcbd5fdbded0 rank 7 nranks 8 cudaDev 3 nvmlDev 3 busId 1901000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-047:237291:237678 [2] NCCL INFO ncclCommInitRank comm 0xc73c5bd41e40 rank 6 nranks 8 cudaDev 2 nvmlDev 2 busId 1801000 commId 0xee13c4f1e7e030dc - Init START slurm-gb200-217-027:234738:235129 [0] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237290:237677 [1] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237291:237678 [2] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237292:237676 [3] NCCL INFO RAS client listening socket at ::1<28028> slurm-gb200-217-047:237291:237678 [2] NCCL INFO Bootstrap timings total 0.001874 (create 0.000049, send 0.000236, recv 0.000857, ring 0.000277, delay 0.000001) slurm-gb200-217-047:237292:237676 [3] NCCL INFO Bootstrap timings total 0.001987 (create 0.000058, send 0.000308, recv 0.000486, ring 0.000298, delay 0.000001) slurm-gb200-217-027:234738:235129 [0] NCCL INFO Bootstrap timings total 0.150981 (create 0.000092, send 0.000167, recv 0.099449, ring 0.000395, delay 0.000001) slurm-gb200-217-027:234739:235132 [1] NCCL INFO Bootstrap timings total 0.051531 (create 0.000051, send 0.000094, recv 0.000164, ring 0.050388, delay 0.000001) slurm-gb200-217-027:234740:235130 [2] NCCL INFO Bootstrap timings total 0.051642 (create 0.000049, send 0.000100, recv 0.000104, ring 0.050388, delay 0.000001) slurm-gb200-217-027:234741:235131 [3] NCCL INFO Bootstrap timings total 0.052601 (create 0.000062, send 0.000132, recv 0.037772, ring 0.013873, delay 0.000001) slurm-gb200-217-047:237290:237677 [1] NCCL INFO Bootstrap timings total 0.018151 (create 0.000077, send 0.000318, recv 0.016776, ring 0.000449, delay 0.000001) slurm-gb200-217-047:237289:237675 [0] NCCL INFO Bootstrap timings total 0.015634 (create 0.000060, send 0.000705, recv 0.000611, ring 0.013624, delay 0.000001) slurm-gb200-217-047:237292:237676 [3] NCCL INFO MNNVL busId 0x1901000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-047:237289:237675 [0] NCCL INFO MNNVL busId 0x801000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-027:234741:235131 [3] NCCL INFO MNNVL busId 0x1901000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-027:234740:235130 [2] NCCL INFO MNNVL busId 0x1801000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-027:234738:235129 [0] NCCL INFO MNNVL busId 0x801000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-047:237290:237677 [1] NCCL INFO MNNVL busId 0x901000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-027:234739:235132 [1] NCCL INFO MNNVL busId 0x901000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-047:237291:237678 [2] NCCL INFO MNNVL busId 0x1801000 fabric UUID 254db2329a6da67a.4641fa3fdb4d7484 cliqueId 0x7ffe state 3 healthMask 0xaa slurm-gb200-217-047:237291:237678 [2] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 6 slurm-gb200-217-047:237289:237675 [0] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 4 slurm-gb200-217-047:237292:237676 [3] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 7 slurm-gb200-217-047:237290:237677 [1] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 5 slurm-gb200-217-027:234741:235131 [3] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 3 slurm-gb200-217-027:234739:235132 [1] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 1 slurm-gb200-217-027:234740:235130 [2] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 2 slurm-gb200-217-027:234738:235129 [0] NCCL INFO MNNVL 1 cliqueId 7ffe cliqueSize 8 cliqueRank 0 slurm-gb200-217-047:237292:237676 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,ffffff00,00000000,00000000 slurm-gb200-217-047:237292:237676 [3] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-047:237292:237676 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-047:237291:237678 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,ffffff00,00000000,00000000 slurm-gb200-217-047:237291:237678 [2] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-047:237291:237678 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-027:234740:235130 [2] NCCL INFO Setting affinity for GPU 2 to ffff,ffffffff,ffffff00,00000000,00000000 slurm-gb200-217-027:234740:235130 [2] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-027:234740:235130 [2] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-027:234741:235131 [3] NCCL INFO Setting affinity for GPU 3 to ffff,ffffffff,ffffff00,00000000,00000000 slurm-gb200-217-027:234741:235131 [3] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-047:237290:237677 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffffffff slurm-gb200-217-027:234741:235131 [3] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-047:237290:237677 [1] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-047:237290:237677 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-027:234739:235132 [1] NCCL INFO Setting affinity for GPU 1 to ff,ffffffff,ffffffff slurm-gb200-217-027:234738:235129 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffffffff slurm-gb200-217-027:234739:235132 [1] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-027:234739:235132 [1] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-047:237289:237675 [0] NCCL INFO Setting affinity for GPU 0 to ff,ffffffff,ffffffff slurm-gb200-217-027:234738:235129 [0] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-027:234738:235129 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-047:237289:237675 [0] NCCL INFO NCCL_COLLNET_ENABLE set by environment to 0. slurm-gb200-217-047:237289:237675 [0] NCCL INFO NCCL_NVLS_ENABLE set by environment to 0. slurm-gb200-217-047:237292:237676 [3] NCCL INFO comm 0xbcbd5fdbded0 rank 7 nRanks 8 nNodes 1 localRanks 8 localRank 7 MNNVL 1 slurm-gb200-217-027:234740:235130 [2] NCCL INFO comm 0xabb25a0df930 rank 2 nRanks 8 nNodes 1 localRanks 8 localRank 2 MNNVL 1 slurm-gb200-217-027:234741:235131 [3] NCCL INFO comm 0xb4ab52367d50 rank 3 nRanks 8 nNodes 1 localRanks 8 localRank 3 MNNVL 1 slurm-gb200-217-047:237291:237678 [2] NCCL INFO comm 0xc73c5bd41e40 rank 6 nRanks 8 nNodes 1 localRanks 8 localRank 6 MNNVL 1 slurm-gb200-217-047:237292:237676 [3] NCCL INFO Trees [0] -1/-1/-1->7->6 [1] -1/-1/-1->7->6 [2] -1/-1/-1->7->6 [3] -1/-1/-1->7->6 [4] -1/-1/-1->7->6 [5] -1/-1/-1->7->6 [6] -1/-1/-1->7->6 [7] -1/-1/-1->7->6 [8] -1/-1/-1->7->6 [9] -1/-1/-1->7->6 [10] -1/-1/-1->7->6 [11] -1/-1/-1->7->6 [12] -1/-1/-1->7->6 [13] -1/-1/-1->7->6 [14] -1/-1/-1->7->6 [15] -1/-1/-1->7->6 [16] -1/-1/-1->7->6 [17] -1/-1/-1->7->6 [18] -1/-1/-1->7->6 [19] -1/-1/-1->7->6 [20] -1/-1/-1->7->6 [21] -1/-1/-1->7->6 [22] -1/-1/-1->7->6 [23] -1/-1/-1->7->6 [24] -1/-1/-1->7->6 [25] -1/-1/-1->7->6 [26] -1/-1/-1->7->6 [27] -1/-1/-1->7->6 [28] -1/-1/-1->7->6 [29] -1/-1/-1->7->6 [30] -1/-1/-1->7->6 [31] -1/-1/-1->7->6 slurm-gb200-217-047:237292:237676 [3] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-027:234738:235129 [0] NCCL INFO comm 0xc4b2133f8d20 rank 0 nRanks 8 nNodes 1 localRanks 8 localRank 0 MNNVL 1 slurm-gb200-217-027:234740:235130 [2] NCCL INFO Trees [0] 3/-1/-1->2->1 [1] 3/-1/-1->2->1 [2] 3/-1/-1->2->1 [3] 3/-1/-1->2->1 [4] 3/-1/-1->2->1 [5] 3/-1/-1->2->1 [6] 3/-1/-1->2->1 [7] 3/-1/-1->2->1 [8] 3/-1/-1->2->1 [9] 3/-1/-1->2->1 [10] 3/-1/-1->2->1 [11] 3/-1/-1->2->1 [12] 3/-1/-1->2->1 [13] 3/-1/-1->2->1 [14] 3/-1/-1->2->1 [15] 3/-1/-1->2->1 [16] 3/-1/-1->2->1 [17] 3/-1/-1->2->1 [18] 3/-1/-1->2->1 [19] 3/-1/-1->2->1 [20] 3/-1/-1->2->1 [21] 3/-1/-1->2->1 [22] 3/-1/-1->2->1 [23] 3/-1/-1->2->1 [24] 3/-1/-1->2->1 [25] 3/-1/-1->2->1 [26] 3/-1/-1->2->1 [27] 3/-1/-1->2->1 [28] 3/-1/-1->2->1 [29] 3/-1/-1->2->1 [30] 3/-1/-1->2->1 [31] 3/-1/-1->2->1 slurm-gb200-217-027:234740:235130 [2] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-027:234741:235131 [3] NCCL INFO Trees [0] 4/-1/-1->3->2 [1] 4/-1/-1->3->2 [2] 4/-1/-1->3->2 [3] 4/-1/-1->3->2 [4] 4/-1/-1->3->2 [5] 4/-1/-1->3->2 [6] 4/-1/-1->3->2 [7] 4/-1/-1->3->2 [8] 4/-1/-1->3->2 [9] 4/-1/-1->3->2 [10] 4/-1/-1->3->2 [11] 4/-1/-1->3->2 [12] 4/-1/-1->3->2 [13] 4/-1/-1->3->2 [14] 4/-1/-1->3->2 [15] 4/-1/-1->3->2 [16] 4/-1/-1->3->2 [17] 4/-1/-1->3->2 [18] 4/-1/-1->3->2 [19] 4/-1/-1->3->2 [20] 4/-1/-1->3->2 [21] 4/-1/-1->3->2 [22] 4/-1/-1->3->2 [23] 4/-1/-1->3->2 [24] 4/-1/-1->3->2 [25] 4/-1/-1->3->2 [26] 4/-1/-1->3->2 [27] 4/-1/-1->3->2 [28] 4/-1/-1->3->2 [29] 4/-1/-1->3->2 [30] 4/-1/-1->3->2 [31] 4/-1/-1->3->2 slurm-gb200-217-027:234741:235131 [3] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 00/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 01/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-047:237291:237678 [2] NCCL INFO Trees [0] 7/-1/-1->6->5 [1] 7/-1/-1->6->5 [2] 7/-1/-1->6->5 [3] 7/-1/-1->6->5 [4] 7/-1/-1->6->5 [5] 7/-1/-1->6->5 [6] 7/-1/-1->6->5 [7] 7/-1/-1->6->5 [8] 7/-1/-1->6->5 [9] 7/-1/-1->6->5 [10] 7/-1/-1->6->5 [11] 7/-1/-1->6->5 [12] 7/-1/-1->6->5 [13] 7/-1/-1->6->5 [14] 7/-1/-1->6->5 [15] 7/-1/-1->6->5 [16] 7/-1/-1->6->5 [17] 7/-1/-1->6->5 [18] 7/-1/-1->6->5 [19] 7/-1/-1->6->5 [20] 7/-1/-1->6->5 [21] 7/-1/-1->6->5 [22] 7/-1/-1->6->5 [23] 7/-1/-1->6->5 [24] 7/-1/-1->6->5 [25] 7/-1/-1->6->5 [26] 7/-1/-1->6->5 [27] 7/-1/-1->6->5 [28] 7/-1/-1->6->5 [29] 7/-1/-1->6->5 [30] 7/-1/-1->6->5 [31] 7/-1/-1->6->5 slurm-gb200-217-047:237291:237678 [2] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-027:234739:235132 [1] NCCL INFO comm 0xbf66c0b18250 rank 1 nRanks 8 nNodes 1 localRanks 8 localRank 1 MNNVL 1 slurm-gb200-217-047:237290:237677 [1] NCCL INFO comm 0xb82cf5a83890 rank 5 nRanks 8 nNodes 1 localRanks 8 localRank 5 MNNVL 1 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 02/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 03/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 04/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 05/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 06/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 07/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-047:237290:237677 [1] NCCL INFO Trees [0] 6/-1/-1->5->4 [1] 6/-1/-1->5->4 [2] 6/-1/-1->5->4 [3] 6/-1/-1->5->4 [4] 6/-1/-1->5->4 [5] 6/-1/-1->5->4 [6] 6/-1/-1->5->4 [7] 6/-1/-1->5->4 [8] 6/-1/-1->5->4 [9] 6/-1/-1->5->4 [10] 6/-1/-1->5->4 [11] 6/-1/-1->5->4 [12] 6/-1/-1->5->4 [13] 6/-1/-1->5->4 [14] 6/-1/-1->5->4 [15] 6/-1/-1->5->4 [16] 6/-1/-1->5->4 [17] 6/-1/-1->5->4 [18] 6/-1/-1->5->4 [19] 6/-1/-1->5->4 [20] 6/-1/-1->5->4 [21] 6/-1/-1->5->4 [22] 6/-1/-1->5->4 [23] 6/-1/-1->5->4 [24] 6/-1/-1->5->4 [25] 6/-1/-1->5->4 [26] 6/-1/-1->5->4 [27] 6/-1/-1->5->4 [28] 6/-1/-1->5->4 [29] 6/-1/-1->5->4 [30] 6/-1/-1->5->4 [31] 6/-1/-1->5->4 slurm-gb200-217-047:237290:237677 [1] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 08/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 09/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 10/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 11/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 12/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 13/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-047:237289:237675 [0] NCCL INFO comm 0xbdcbb5305fc0 rank 4 nRanks 8 nNodes 1 localRanks 8 localRank 4 MNNVL 1 slurm-gb200-217-027:234739:235132 [1] NCCL INFO Trees [0] 2/-1/-1->1->0 [1] 2/-1/-1->1->0 [2] 2/-1/-1->1->0 [3] 2/-1/-1->1->0 [4] 2/-1/-1->1->0 [5] 2/-1/-1->1->0 [6] 2/-1/-1->1->0 [7] 2/-1/-1->1->0 [8] 2/-1/-1->1->0 [9] 2/-1/-1->1->0 [10] 2/-1/-1->1->0 [11] 2/-1/-1->1->0 [12] 2/-1/-1->1->0 [13] 2/-1/-1->1->0 [14] 2/-1/-1->1->0 [15] 2/-1/-1->1->0 [16] 2/-1/-1->1->0 [17] 2/-1/-1->1->0 [18] 2/-1/-1->1->0 [19] 2/-1/-1->1->0 [20] 2/-1/-1->1->0 [21] 2/-1/-1->1->0 [22] 2/-1/-1->1->0 [23] 2/-1/-1->1->0 [24] 2/-1/-1->1->0 [25] 2/-1/-1->1->0 [26] 2/-1/-1->1->0 [27] 2/-1/-1->1->0 [28] 2/-1/-1->1->0 [29] 2/-1/-1->1->0 [30] 2/-1/-1->1->0 [31] 2/-1/-1->1->0 slurm-gb200-217-027:234739:235132 [1] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-047:237289:237675 [0] NCCL INFO Trees [0] 5/-1/-1->4->3 [1] 5/-1/-1->4->3 [2] 5/-1/-1->4->3 [3] 5/-1/-1->4->3 [4] 5/-1/-1->4->3 [5] 5/-1/-1->4->3 [6] 5/-1/-1->4->3 [7] 5/-1/-1->4->3 [8] 5/-1/-1->4->3 [9] 5/-1/-1->4->3 [10] 5/-1/-1->4->3 [11] 5/-1/-1->4->3 [12] 5/-1/-1->4->3 [13] 5/-1/-1->4->3 [14] 5/-1/-1->4->3 [15] 5/-1/-1->4->3 [16] 5/-1/-1->4->3 [17] 5/-1/-1->4->3 [18] 5/-1/-1->4->3 [19] 5/-1/-1->4->3 [20] 5/-1/-1->4->3 [21] 5/-1/-1->4->3 [22] 5/-1/-1->4->3 [23] 5/-1/-1->4->3 [24] 5/-1/-1->4->3 [25] 5/-1/-1->4->3 [26] 5/-1/-1->4->3 [27] 5/-1/-1->4->3 [28] 5/-1/-1->4->3 [29] 5/-1/-1->4->3 [30] 5/-1/-1->4->3 [31] 5/-1/-1->4->3 slurm-gb200-217-047:237289:237675 [0] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 14/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 15/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 16/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 17/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 18/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 19/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 20/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 21/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 22/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-047:237291:237700 [2] NCCL INFO [Proxy Service] Device 2 CPU core 140 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 23/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 24/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 25/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 26/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 27/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 28/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-047:237292:237699 [3] NCCL INFO [Proxy Service] Device 3 CPU core 79 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 29/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 30/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Channel 31/32 : 0 1 2 3 4 5 6 7 slurm-gb200-217-047:237291:237702 [2] NCCL INFO [Proxy Service UDS] Device 2 CPU core 73 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 [4] 1/-1/-1->0->-1 [5] 1/-1/-1->0->-1 [6] 1/-1/-1->0->-1 [7] 1/-1/-1->0->-1 [8] 1/-1/-1->0->-1 [9] 1/-1/-1->0->-1 [10] 1/-1/-1->0->-1 [11] 1/-1/-1->0->-1 [12] 1/-1/-1->0->-1 [13] 1/-1/-1->0->-1 [14] 1/-1/-1->0->-1 [15] 1/-1/-1->0->-1 [16] 1/-1/-1->0->-1 [17] 1/-1/-1->0->-1 [18] 1/-1/-1->0->-1 [19] 1/-1/-1->0->-1 [20] 1/-1/-1->0->-1 [21] 1/-1/-1->0->-1 [22] 1/-1/-1->0->-1 [23] 1/-1/-1->0->-1 [24] 1/-1/-1->0->-1 [25] 1/-1/-1->0->-1 [26] 1/-1/-1->0->-1 [27] 1/-1/-1->0->-1 [28] 1/-1/-1->0->-1 [29] 1/-1/-1->0->-1 [30] 1/-1/-1->0->-1 [31] 1/-1/-1->0->-1 slurm-gb200-217-027:234738:235129 [0] NCCL INFO P2P Chunksize set to 524288 slurm-gb200-217-047:237292:237701 [3] NCCL INFO [Proxy Service UDS] Device 3 CPU core 80 slurm-gb200-217-027:234738:235129 [0] NCCL INFO Check P2P Type intraNodeP2pSupport 1 directMode 0 slurm-gb200-217-027:234740:235153 [2] NCCL INFO [Proxy Service] Device 2 CPU core 112 slurm-gb200-217-027:234741:235154 [3] NCCL INFO [Proxy Service] Device 3 CPU core 126 slurm-gb200-217-027:234741:235155 [3] NCCL INFO [Proxy Service UDS] Device 3 CPU core 129 slurm-gb200-217-027:234740:235156 [2] NCCL INFO [Proxy Service UDS] Device 2 CPU core 114 slurm-gb200-217-047:237289:237704 [0] NCCL INFO [Proxy Service] Device 0 CPU core 4 slurm-gb200-217-047:237290:237706 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 60 slurm-gb200-217-027:234739:235157 [1] NCCL INFO [Proxy Service] Device 1 CPU core 8 slurm-gb200-217-047:237290:237703 [1] NCCL INFO [Proxy Service] Device 1 CPU core 2 slurm-gb200-217-027:234738:235158 [0] NCCL INFO [Proxy Service] Device 0 CPU core 8 slurm-gb200-217-047:237289:237705 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 6 slurm-gb200-217-027:234739:235159 [1] NCCL INFO [Proxy Service UDS] Device 1 CPU core 45 slurm-gb200-217-027:234738:235160 [0] NCCL INFO [Proxy Service UDS] Device 0 CPU core 45 slurm-gb200-217-027:234741:235131 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-027:234741:235131 [3] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-047:237291:237678 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-047:237291:237678 [2] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-027:234739:235132 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-027:234739:235132 [1] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-047:237292:237676 [3] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-047:237292:237676 [3] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-027:234740:235130 [2] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-027:234740:235130 [2] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-047:237290:237677 [1] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-047:237290:237677 [1] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-047:237289:237675 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-047:237289:237675 [0] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-027:234738:235129 [0] NCCL INFO threadThresholds 8/8/64 | 64/8/64 | 512 | 512 slurm-gb200-217-027:234738:235129 [0] NCCL INFO 32 coll channels, 32 collnet channels, 0 nvls channels, 32 p2p channels, 32 p2p channels per peer slurm-gb200-217-027:234738:235129 [0] NCCL INFO CC Off, workFifoBytes 1048576 slurm-gb200-217-027:234740:235130 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-027:234740:235130 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-027:234740:235130 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-027:234740:235130 [2] NCCL INFO ncclCommInitRank comm 0xabb25a0df930 rank 2 nranks 8 cudaDev 2 nvmlDev 2 busId 1801000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-027:234740:235130 [2] NCCL INFO Init timings - ncclCommInitRank: rank 2 nranks 8 total 0.94 (kernels 0.09, alloc 0.16, bootstrap 0.05, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.01) slurm-gb200-217-027:234741:235131 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-027:234741:235131 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-027:234741:235131 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-027:234741:235131 [3] NCCL INFO ncclCommInitRank comm 0xb4ab52367d50 rank 3 nranks 8 cudaDev 3 nvmlDev 3 busId 1901000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-027:234741:235131 [3] NCCL INFO Init timings - ncclCommInitRank: rank 3 nranks 8 total 0.93 (kernels 0.09, alloc 0.16, bootstrap 0.05, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.08, rest 0.01) slurm-gb200-217-047:237291:237678 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-047:237291:237678 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-047:237292:237676 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-047:237292:237676 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-047:237292:237676 [3] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-047:237291:237678 [2] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-047:237291:237678 [2] NCCL INFO ncclCommInitRank comm 0xc73c5bd41e40 rank 6 nranks 8 cudaDev 2 nvmlDev 2 busId 1801000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-047:237291:237678 [2] NCCL INFO Init timings - ncclCommInitRank: rank 6 nranks 8 total 0.88 (kernels 0.10, alloc 0.14, bootstrap 0.00, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.01) slurm-gb200-217-027:234739:235132 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-027:234739:235132 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-027:234739:235132 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-047:237292:237676 [3] NCCL INFO ncclCommInitRank comm 0xbcbd5fdbded0 rank 7 nranks 8 cudaDev 3 nvmlDev 3 busId 1901000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-047:237292:237676 [3] NCCL INFO Init timings - ncclCommInitRank: rank 7 nranks 8 total 0.88 (kernels 0.09, alloc 0.16, bootstrap 0.00, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.01) slurm-gb200-217-027:234739:235132 [1] NCCL INFO ncclCommInitRank comm 0xbf66c0b18250 rank 1 nranks 8 cudaDev 1 nvmlDev 1 busId 901000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-027:234739:235132 [1] NCCL INFO Init timings - ncclCommInitRank: rank 1 nranks 8 total 0.93 (kernels 0.09, alloc 0.15, bootstrap 0.05, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.01) slurm-gb200-217-047:237289:237675 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-027:234738:235129 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-027:234738:235129 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-027:234738:235129 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-047:237290:237677 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v4 symbol. slurm-gb200-217-047:237290:237677 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-027:234738:235129 [0] NCCL INFO ncclCommInitRank comm 0xc4b2133f8d20 rank 0 nranks 8 cudaDev 0 nvmlDev 0 busId 801000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-047:237289:237675 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v3 symbol. slurm-gb200-217-047:237289:237675 [0] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-047:237289:237675 [0] NCCL INFO ncclCommInitRank comm 0xbdcbb5305fc0 rank 4 nranks 8 cudaDev 0 nvmlDev 0 busId 801000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-027:234738:235129 [0] NCCL INFO Init timings - ncclCommInitRank: rank 0 nranks 8 total 0.97 (kernels 0.09, alloc 0.10, bootstrap 0.15, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.00) slurm-gb200-217-047:237290:237677 [1] NCCL INFO TUNER/Plugin: Failed to find ncclTunerPlugin_v2 symbol, using internal tuner instead. slurm-gb200-217-047:237290:237677 [1] NCCL INFO ncclCommInitRank comm 0xb82cf5a83890 rank 5 nranks 8 cudaDev 1 nvmlDev 1 busId 901000 commId 0xee13c4f1e7e030dc - Init COMPLETE slurm-gb200-217-047:237290:237677 [1] NCCL INFO Init timings - ncclCommInitRank: rank 5 nranks 8 total 0.88 (kernels 0.09, alloc 0.14, bootstrap 0.02, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.01) slurm-gb200-217-047:237289:237675 [0] NCCL INFO Init timings - ncclCommInitRank: rank 4 nranks 8 total 0.89 (kernels 0.09, alloc 0.15, bootstrap 0.02, allgathers 0.00, topo 0.52, graphs 0.01, connections 0.09, rest 0.00) # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 00/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 00/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 01/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 01/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 02/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 02/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 03/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 03/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 04/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 04/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 05/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 05/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 06/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 06/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 07/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 07/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 08/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 08/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 09/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 09/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 10/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 10/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 11/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 11/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 12/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 12/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 13/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 13/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 14/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 14/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 15/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 15/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 16/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 16/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 17/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 17/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 18/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 18/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 19/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 19/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 20/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 20/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 21/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 21/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 22/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 22/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 23/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 23/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 24/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 24/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 25/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 25/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 26/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 26/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 27/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 27/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 28/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 28/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 29/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 29/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 30/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 30/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-027:234740:235162 [2] NCCL INFO Channel 31/0 : 2[2] -> 3[3] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] NCCL INFO Channel 31/0 : 3[3] -> 4[0] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 00/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 00/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 00/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 01/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 01/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 01/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 02/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 02/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 03/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 03/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 02/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 00/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 01/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 04/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 04/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 03/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 02/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 05/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 04/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 05/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 03/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 05/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 06/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 06/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 04/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 07/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 07/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 06/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 05/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 07/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 08/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 08/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 06/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 09/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 09/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 07/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 08/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 10/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 09/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 08/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 10/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 10/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 09/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 11/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 11/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 12/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 10/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 11/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 12/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 11/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 13/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 12/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 12/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 13/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 14/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 13/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 15/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 14/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 14/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 13/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 15/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 16/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 14/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 15/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 16/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 17/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 16/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 15/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 17/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 17/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 18/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 16/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 18/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 18/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 19/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 19/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 17/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 19/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 18/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 20/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 20/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 20/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 21/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 19/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 21/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 21/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 20/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 22/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 22/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 23/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 22/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 21/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 23/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 23/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 24/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 22/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 24/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 25/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 24/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 23/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 25/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 24/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 25/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 26/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 26/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 25/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 26/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 27/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 27/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 26/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 27/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 28/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 27/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 28/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 29/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 28/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 29/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 28/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 29/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 30/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 29/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 30/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 30/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 30/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-047:237292:237707 [3] NCCL INFO Channel 31/0 : 7[3] -> 0[0] via P2P/MNNVL slurm-gb200-217-047:237290:237710 [1] NCCL INFO Channel 31/0 : 5[1] -> 6[2] via P2P/MNNVL slurm-gb200-217-047:237289:237709 [0] NCCL INFO Channel 31/0 : 4[0] -> 5[1] via P2P/MNNVL slurm-gb200-217-047:237291:237708 [2] NCCL INFO Channel 31/0 : 6[2] -> 7[3] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 00/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 01/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 02/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 03/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 04/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 00/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 05/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 01/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 06/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 02/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 07/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 03/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 08/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 04/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 09/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 05/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 10/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 06/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 11/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 07/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 12/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 08/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 13/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 09/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 14/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 10/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 15/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 11/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 16/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 12/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 17/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 13/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 18/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 14/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 15/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 19/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 16/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 20/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 17/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 21/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 18/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 22/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 19/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 23/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 20/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 24/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 21/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 25/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 22/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 26/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 23/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 27/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 24/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 28/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 25/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 29/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 26/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 27/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 30/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 28/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234738:235163 [0] NCCL INFO Channel 31/0 : 0[0] -> 1[1] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 29/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 30/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234739:235164 [1] NCCL INFO Channel 31/0 : 1[1] -> 2[2] via P2P/MNNVL slurm-gb200-217-027:234741:235161 [3] transport/p2p.cc:277 NCCL WARN Cuda failure 400 'invalid resource handle' slurm-gb200-217-027:234741:235161 [3] NCCL INFO transport/p2p.cc:352 -> 1 slurm-gb200-217-027:234741:235161 [3] NCCL INFO transport/p2p.cc:487 -> 1 slurm-gb200-217-027:234741:235161 [3] NCCL INFO transport.cc:194 -> 1 slurm-gb200-217-027:234741:235161 [3] NCCL INFO transport/generic.cc:19 -> 1 slurm-gb200-217-027:234741:235161 [3] NCCL INFO group.cc:148 -> 1 slurm-gb200-217-027:234741:235161 [3] NCCL INFO group.cc:75 -> 1 [Async thread] slurm-gb200-217-027:234741:234741 [3] NCCL INFO group.cc:454 -> 1 slurm-gb200-217-027:234741:234741 [3] NCCL INFO group.cc:573 -> 1 slurm-gb200-217-027:234741:234741 [3] NCCL INFO enqueue.cc:2229 -> 1 slurm-gb200-217-027: Test NCCL failure all_reduce.cu:44 'unhandled cuda error (run with NCCL_DEBUG=INFO for details) / ' .. slurm-gb200-217-027 pid 234741: Test failure common.cu:377 .. slurm-gb200-217-027 pid 234741: Test failure common.cu:584 .. slurm-gb200-217-027 pid 234741: Test failure all_reduce.cu:90 .. slurm-gb200-217-027 pid 234741: Test failure common.cu:613 .. slurm-gb200-217-027 pid 234741: Test failure common.cu:1016 .. slurm-gb200-217-027 pid 234741: Test failure common.cu:842 srun: error: slurm-gb200-217-027: task 3: Exited with exit code 3 srun: Terminating StepId=1742.0 slurmstepd: error: *** STEP 1742.0 ON slurm-gb200-217-027 CANCELLED AT 2025-07-14T20:26:11 *** slurmstepd: error: mpi/pmix_v4: _errhandler: slurm-gb200-217-027 [0]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.1742.0:3] srun: Job step aborted: Waiting up to 32 seconds for job step to finish. slurmstepd: error: mpi/pmix_v4: _errhandler: slurm-gb200-217-047 [1]: pmixp_client_v2.c:211: Error handler invoked: status = -61, source = [slurm.pmix.1742.0:6] srun: error: slurm-gb200-217-027: tasks 0-2: Terminated srun: error: slurm-gb200-217-047: tasks 4-7: Terminated
最新发布
07-16
在使用 SLURM 进行多节点 GPU 通信时，若遇到 NCCL 报错 `Cuda failure 400: invalid resource handle`，通常表明某个 GPU 资源句柄无效或访问异常。此类错误可能由多种因素引起，包括但不限于 CUDA 环境配置不当、GPU 资源分配冲突、NCCL 版本与 CUDA 不兼容、或 SLURM 配置未正确绑定 GPU 资源。 以下为可能的原因及解决办法： - **资源句柄无效**：该错误代码 `400` 表示“invalid resource handle”，意味着尝试访问的 CUDA 资源（如事件、流、内存等）不存在或已被释放。建议检查程序中是否重复释放了某些资源，或者在多个线程/进程中错误地共享了资源句柄 [^2]。 - **SLURM GPU 绑定问题**：确保在提交作业时通过 `--gpus-per-node` 或 `--gres=gpu:N` 正确指定了使用的 GPU 数量，并且在程序中通过 `CUDA_VISIBLE_DEVICES` 设置了正确的设备索引。例如： ```bash #SBATCH --gres=gpu:gb200:4 export CUDA_VISIBLE_DEVICES=0,1,2,3 ``` 若未正确限制可见设备，可能导致进程试图访问不属于当前任务的 GPU 设备 [^1]。 - **NCCL 和 CUDA 版本兼容性**：确保所使用的 NCCL 版本与 CUDA 工具包版本兼容。可参考 NVIDIA 官方文档确认兼容性列表。若版本不匹配，可能出现资源管理异常。构建 NCCL 时建议指定架构参数以适配 GB200 的计算能力，例如： ```bash export NVCC_GENCODE="-gencode=arch=compute_90,code=sm_90" make -j CUDA_HOME=/usr/local/cuda ``` - **多节点通信初始化失败**：在分布式训练中，NCCL 需要通过 SSH 或 MPI 正确初始化跨节点通信。确保节点间网络通畅，并使用 `nccl-tests` 或 `torchrun` / `mpiexec` 启动多节点任务时配置正确。可以先运行简单的 `all_reduce_perf` 测试验证基本通信能力 [^1]。 - **驱动和运行时状态异常**：使用 `nvidia-smi` 检查所有目标 GPU 是否处于正常状态，无显存泄漏或被其他进程占用。若存在僵尸进程，可通过 `kill -9 <pid>` 清理 [^3]。 ### 示例命令用于测试 NCCL 多节点通信 ```bash srun -N2 -n4 --gpus-per-node=4 --gpu-bind=closest ./build/all_reduce_perf -b 8 -e 512M -f 2 -g 1 ``` 上述命令表示在两个节点上各启动 4 个 GPU 进程，进行 AllReduce 性能测试，并绑定最近的 GPU 设备。 ---
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值