RDMA在KVM实现条件

KVM 支持VF passthrough条件

CPU必须支持 Intel VT-d 或 AMD-Vi（IOMMU）技术

demsg要包含下述两部分

• DMAR: Intel(R) Virtualization Technology for Directed I/O
• DMAR: IOMMU enabled

检查CPU是否支持VT-d或AMD-Vi

# dmesg |grep -e "DMAR" -e "IOMMU"|grep -e "Virtualization" -e enabled [    0.000000] DMAR: IOMMU enabled [    0.001068] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping. [    1.150702] DMAR: Intel(R) Virtualization Technology for Directed I/O

内核必须支持vfio, vfio_iommu_type1, vfio_pci 等模块

检查Kernel加载 IOMMU 相关的内核模块

[root@stgExt1 qemu]# lsmod|grep -e vfio -e iommu vfio_pci               61440  0 vfio_virqfd            16384  1 vfio_pci vfio_iommu_type1       36864  0 vfio                   36864  2 vfio_iommu_type1,vfio_pci irqbypass              16384  422 vfio_pci,kvm

QEMU必须2.0版本以上

centos8.4自带qemu版本4.2.0，BVT环境已升级至8.0.2，且QEMU需要重新编译

configure ./

./configure --prefix=/usr/local/qemu_rdma/ --enable-debug --enable-kvm --enable-vnc --target-list=x86_64-softmmu --enable-spice --enable-spice-protocol --enable-vnc --enable-usb-redir --enable-rdma

QEMU替换步骤

example

ln -sf /usr/local/qemu_rdma/bin/qemu-system-x86_64 /usr/libexec/qemu-kvm setenforce 0

libvirt 版本是 1.2.9 或更高版本

centos8.4自带libvirt 版本为6.0.0

KVM支持SR-IOV

我们把SR-IOV创建出的虚拟网卡称为VF，如下命令可以查看网卡物理端口ens4f0/1（称PF）最大支持创建的VF均为8个；

KVM支持SR-IOV

我们把SR-IOV创建出的虚拟网卡称为VF，如下命令可以查看网卡物理端口ens4f0/1（称PF）最大支持创建的VF均为8个；

# cat /sys/class/net/ens4f0/device/sriov_totalvfs 8 # cat /sys/class/net/ens4f1/device/sriov_totalvfs 8

ens4f0单个网口虚拟出6个VF

# echo 6 > /sys/class/net/ens4f0/device/sriov_numvfs # lspci|grep Mellanox b1:00.0 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] b1:00.1 Ethernet controller: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] b1:00.2 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function b1:00.3 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function b1:00.4 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function b1:00.5 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function b1:00.6 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function b1:00.7 Ethernet controller: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function # ip link |grep ens4 261: ens4f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 262: ens4f0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 263: ens4f0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 264: ens4f0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 265: ens4f0v4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 266: ens4f0v5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 18: ens4f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 19: ens4f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 # ip link show ens4f0v0 261: ens4f0v0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000     link/ether 56:ba:79:b5:fb:3a brd ff:ff:ff:ff:ff:ff [root@stgExt1 qemu]# ip link show ens4f0v1 262: ens4f0v1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000     link/ether 42:f9:c8:62:be:fd brd ff:ff:ff:ff:ff:ff [root@stgExt1 qemu]# ip link show ens4f0v2 263: ens4f0v2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000     link/ether 2e:2b:21:22:a7:da brd ff:ff:ff:ff:ff:ff [root@stgExt1 qemu]# ip link show ens4f0v3 264: ens4f0v3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000     link/ether 22:cd:f8:8e:8b:39 brd ff:ff:ff:ff:ff:ff [root@stgExt1 qemu]# ip link show ens4f0v4 265: ens4f0v4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000     link/ether b6:b1:22:d5:28:46 brd ff:ff:ff:ff:ff:ff [root@stgExt1 qemu]# ip link show ens4f0v5 266: ens4f0v5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000     link/ether be:64:4f:36:e0:f7 brd ff:ff:ff:ff:ff:ff

lspci命令行输出

# lspci -nn |grep Mellanox b1:00.0 Ethernet controller [0200]: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] [15b3:101f] b1:00.1 Ethernet controller [0200]: Mellanox Technologies MT2894 Family [ConnectX-6 Lx] [15b3:101f] b1:00.2 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] b1:00.3 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] b1:00.4 Ethernet controller [0200]: Mellanox Technologies ConnectX Family mlx5Gen Virtual Function [15b3:101e] b1:00.5 Ethernet controller [<