【RDMA】intel 英特尔RDMA 驱动和ibverslib 库安装|流控PFC

最新推荐文章于 2025-02-12 21:52:52 发布

bdview

最新推荐文章于 2025-02-12 21:52:52 发布

阅读量1.6k

点赞数

文章标签： linux 运维服务器

本文链接：https://blog.youkuaiyun.com/weixin_42319496/article/details/125942759

版权

本文详细介绍了在Linux服务器上安装Intel RDMA驱动和ibverbs库的步骤，包括检查已安装版本、编译安装、设置iWARP/RoCEv2、流控PFC、ECN配置等关键环节。此外，还涵盖了错误处理、IP配置、查看设备与网口对应关系等内容，为RDMA功能的确认和优化提供了全面指导。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

关键步骤概览

查已经安装的版本

详细步骤和说明

编译和安装(Building and Installation )

确认RDMA功能(Confirm RDMA Functionality)

iWARP / RoCEv2选择(iWARP/RoCEv2 Selection)

FC|PFC流控制设置(Flow Control Settings)

作者：bandaoyu 地址：https://blog.youkuaiyun.com/bandaoyu/article/details/116203690

更新：

intel 已经提供驱动完整包，包含ice lan驱动和irdma内核态rdma驱动，无需分开下载：英特尔® 以太网适配器完整的驱动程序包

~~1、安装固件，固件在NVMUpdatePackage文件夹~~

2、安装lan驱动(ice)，在PROCGB文件夹

3、安装rdma内核驱动(irdma)，在RDMA文件夹(装了irdma后,ibv_devinfo才有效)

4、安装rdma用户库(rdma-core)，

5、安装固件，固件在NVMUpdatePackage文件夹(发现安装了ice (可能是ice+irdma)重新之后，才能装固件)

sudo ./nvmupdate64e -u -l -o update.xml -b -c nvmupdate.cfg

----------原教程：----------------------------------------------------

关键步骤概览

(先装ice-->装irdma-->装rdma-core)

1、服务器上安装intel RDMA物理网卡

2、到官网下载驱动适用于 E810 和 X722 英特尔® 以太网控制器的 Linux* RDMA 驱动程序

3、安装相应的 LAN 驱动程序 (在安装 irdma 之前， E810 和 X722 对应的 LAN 驱动程序(ice 或 i40e)都必须从此版本中包含的源代码构建安装在您的系统上。)(在intel 官网搜索ICE)搜索- Intel.com. 如：E810：适用于 E810 系列设备的英特尔® 网络适配器驱动程序 Linux* 下

下载ICE包，解压，(看readme，按readme操作)进入src，make install。(如果因为环境问题安装失败了，改变环境后再试之前，请make clean)

4、安装irdma

(irdma Linux *驱动程序在支持RDMA的英特尔网络设备上启用RDMA功能。)

5、安装固件(如果需要的话)

6、安装依赖(见本文的“报错处理”)安装rdma-core (用户空间的ibvers库，给应用程序编程提供接口)

注意，执行 patch -p2 < /path/to/irdma-/libirdma-27.0.patch 命令时，别落了“<”符号

7、设置网卡驱动加载模式iWARP或RoCEv2

ibv_devinfo命令查看网卡模式

transport: iWARP (1)

查已经安装的版本

查ice modinfo ice

[SDS_Admin@rdma65 data]$ modinfo ice
filename: /lib/modules/5.10.38-21.hl05.el7.x86_64/updates/drivers/net/ethernet/intel/ice/ice.ko
firmware: intel/ice/ddp/ice.pkg
version: 1.7.16
license: GPL v2

查irdma modinfo irdma

[SDS_Admin@rdma65 data]$ modinfo irdma
filename: /lib/modules/5.10.38-21.hl05.el7.x86_64/updates/drivers/infiniband/hw/irdma/irdma.ko
version: 1.7.72
license: Dual BSD/GPL
description: Intel(R) Ethernet Protocol Driver for RDMA

查ib-core modinfo ib-core

[SDS_Admin@rdma65 data]$ modinfo ib-core
filename: /lib/modules/5.10.38-21.hl05.el7.x86_64/kernel/drivers/infiniband/core/ib_core.ko.xz
alias: rdma-netlink-subsys-4
license: Dual BSD/GPL
description: core kernel InfiniBand API

详细步骤和说明

原文：readme.txt:https://downloadmirror.intel.com/30368/eng/README_irdma_1.4.22.txt

==============================================================================
irdma - Linux* RDMA Driver for the E810 and X722 Intel(R) Ethernet Controllers
==============================================================================

--------
目录
--------
- Overview 概览
- Prerequisites 前提和依赖
- Supported OS List 支持是操作系统
- Building and Installation 编译和安装
- Confirm RDMA Functionality 确认RDMA功能
- iWARP/RoCEv2 Selection 选择iWARP或RoCEv2
- iWARP Port Mapper (iwpmd)
- Flow Control Settings
- ECN Configuration
- Devlink Configuration
- Memory Requirements 内存需求
- Resource Profile Limits 资源限制
- Resource Limits Selector 资源限制选择
- RDMA Statistics RDMA统计
- perftest 性能测试工具
- MPI
- Performance
- Interoperability
- Dynamic Tracing
- Dynamic Debug
- Capturing RDMA Traffic with tcpdump
- Known Issues/Notes

--------
概览(Overview)
--------

irdma Linux*驱动程序使支持RDMA的英特尔网络设备具有RDMA功能。

此驱动程序支持的设备：

The irdma Linux* driver enables RDMA functionality on RDMA-capable Intel
network devices. Devices supported by this driver:
- Intel(R) Ethernet Controller E810
- Intel(R) Ethernet Network Adapter X722

E810和X722设备各自支持一组不同的RDMA功能。

The E810 and X722 devices each support a different set of RDMA features.

-E810支持iWARP和RoCEv2 RDMA传输，还支持优先流控制(PFC)和

显式拥塞通知(ECN)。

-X722仅支持iWARP和一组更有限的配置参数。

- E810 supports both iWARP and RoCEv2 RDMA transports, and also supports
congestion management features like priority flow control (PFC) and
explicit congestion notification (ECN).
- X722 supports only iWARP and a more limited set of configuration
parameters.

本文档的每个部分都描述了适配器之间的差异。

对于E810和X722，必须在安装irdma之前安装相应的LAN驱动程序(ice或i40e)，从本版本中包含的源代码构建，安装在您的系统上。

Differences between adapters are described in each section of this document.

For both E810 and X722, the corresponding LAN driver (ice or i40e) must be
built from source included in this release and installed on your system prior
to installing irdma.

-------------
先决条件(Prerequisites)
-------------

- Compile and install the E810 or X722 LAN PF driver from source included in
this release. Refer to the ice or i40e driver README for installation
instructions.
* For E810 adapters, use the ice driver.
* For X722 adapters, use the i40e driver.
- For best results, use a fully supported OS from the Supported OS List below.
- For server memory requirements, see the "Memory Requirements" section of this
document.
- Install required packages. Refer to the "Building" section of the rdma-core

README for required packages for your OS:
https://github.com/linux-rdma/rdma-core/blob/v27.0/README.md
* RHEL 7 and SLES:
Install all required packages listed in the rdma-core README.
* RHEL 8:
Install the required packages for RHEL 7, then install the following
additional packages:
dnf install python3-docutils perl-generators
* Ubuntu:
Install the required packages listed in the rdma-core README, then
install the following additional package:
apt-get install python3-docutils libsystemd-dev

* Note:

以下是可用于获取 rdma-core依赖包的repo文件示例，当然，这些可能不是所需的全部。

The following are sample repo files that can be used to get the dependent packages
for rdma-core. However, these may not be all that is required.

- For SLES
http://download.opensuse.org/distribution/leap/42.3/repo/oss

- For RHEL 8.1
http://vault.centos.org/8.1.1911/PowerTools/x86_64/os/

-----------------
支持的操作系统列表(Supported OS List )
-----------------

Supported:
* RHEL 8.3
* RHEL 7.9
* SLES 15 SP2
* SLES 12 SP5
* Ubuntu 18.04
* Ubuntu 20.04

Supported Not Validated:
* RHEL 8.2
* RHEL 8.1
* RHEL 8
* RHEL 7.8
* RHEL 7.7
* RHEL 7.6 + OFED 4.17-1
* RHEL 7.5 + OFED 4.17-1
* RHEL 7.4 + OFED 4.17-1
* SLES 15 SP1
* SLES 15 + OFED 4.17-1
* SLES 12 SP 4 + OFED 4.17-1
* SLES 12 SP 3 + OFED 4.17-1
* Linux kernel stable 5.10.*
* Linux kernel longterm 5.4.*, 4.19.*, 4.14.*

-------------------------

编译和安装(Building and Installation )

-------------------------

To build and install the irdma driver and supporting rdma-core libraries:

1.解压缩irdma驱动程序压缩包：
tar zxf irdma-.tgz

2.构建并安装RDMA驱动程序：
cd irdma-
./build.sh

By default, the irdma driver is built using in-distro RDMA libraries and
modules. Optionally, irdma may also be built using OFED modules. See the
Supported OS List above for a list of OSes that support this option.
* Note: Intel products are not validated on other vendors' proprietary
software packages.
To install irdma using OFED modules:
- Download OFED-4.17-1.tgz from the OpenFabrics Alliance:
wget http://openfabrics.org/downloads/OFED/ofed-4.17-1/OFED-4.17-1.tgz
- Decompress the archive:
tar xzvf OFED-4.17.1.tgz
- Install OFED:
cd OFED-4.17-1
./install --all
- Reboot after installation is complete.
- Build the irdma driver with the "ofed" option:
cd /path/to/irdma-
./build.sh ofed
- Continue with the installation steps below.

3.加载驱动程序：
RHEL and Ubuntu:
modprobe irdma

先卸载之前的再加载：

rmmod irdma; modprobe irdma

SLES:
modprobe irdma --allow-unsupported

Notes:
- This modprobe step is required only during installation. Normally,
irdma is autoloaded via a udev rule when ice or i40e is loaded:
/usr/lib/udev/rules.d/90-rdma-hw-modules.rules
- For SLES, to automatically allow loading unsupported modules, add the
following to /etc/modprobe.d/10-unsupported-modules.conf:
allow_unsupported_modules 1

4.卸载任何先前版本的rdma-core用户空间库。
For example, in RHEL:
yum erase rdma-core

Note: "yum erase rdma-core" will also remove any packages that depend on
rdma-core, such as perftest or fio. Please re-install them after
installing rdma-core.

5.Patch，构建和安装rdma-core用户空间库：

RHEL:

安装依赖：
sudo yum install rpm-build -y &&sudo yum install cmake  -y &&sudo yum install libudev-devel -y &&sudo  yum install libnl3-devel -y &&sudo yum install python-docutils -y &&sudo yum install -y valgrind-devel
#1 从GitHub下载rdma-core-27.0.tar.gz
wget https://github.com/linux-rdma/rdma-core/releases/download/v27.0/rdma-core-27.0.tar.gz #2 Apply patch libirdma-27.0.patch to rdma-core
tar -xzvf rdma-core-27.0.tar.gz
cd rdma-core-27.0
patch -p2 < /path/to/irdma-/libirdma-27.0.patch #别落了“<” 符号 #3 确保目录rdma-core / redhat和contents 位于“ root”组下
cd ..
chgrp -R root rdma-core-27.0/redhat #4 重新打包成适当的名称给building用 ( "tgz" 扩展名代替 "tar.gz")
tar -zcvf rdma-core-27.0.tgz rdma-core-27.0 #5 构建 rdma-core
mkdir -p ~/rpmbuild/SOURCES
mkdir -p ~/rpmbuild/SPECS
cp rdma-core-27.0.tgz ~/rpmbuild/SOURCES/
cd ~/rpmbuild/SOURCES
tar -xzvf rdma-core-27.0.tgz
cp ~/rpmbuild/SOURCES/rdma-core-27.0/redhat/rdma-core.spec ~/rpmbuild/SPECS/
cd ~/rpmbuild/SPECS/
rpmbuild -ba rdma-core.spec #6 安装RPMs
cd ~/rpmbuild/RPMS/x86_64
yum install *27.0*.rpm
VERSION=27.0&&cd ..&&chgrp -R root rdma-core-${VERSION}/redhat&&tar -zcvf rdma-core-${VERSION}.tgz rdma-core-${VERSION}&&mkdir -p ~/rpmbuild/SOURCES&&mkdir -p ~/rpmbuild/SPECS&&cp rdma-core-${VERSION}.tgz ~/rpmbuild/SOURCES/&&cd ~/rpmbuild/SOURCES&&tar -xzvf rdma-core-${VERSION}.tgz&&cp ~/rpmbuild/SOURCES/rdma-core-${VERSION}/

最低0.47元/天解锁文章