【dpdk】qat-getting-started-guide-with-dpdk-v2.0

参考文档

632506-qat-getting-started-guide-v2.0.pdf

21. Intel® QuickAssist (QAT) Crypto Poll Mode Driver — Data Plane Development Kit 23.11.0 documentation (dpdk.org)

1.QAT driver installation

This section provides details on building and installing the software.

1.1. Installation using kernel.org driver

The examples below are based on the C62x device, if you have a different device use the corresponding values in the above table.

In BIOS ensure that SRIOV is enabled and either:

• Disable VT-d or
• Enable VT-d and set "intel_iommu=on iommu=pt" in the grub file.

Check that the QAT driver is loaded on your system, by executing:

lsmod | grep qa

You should see the kernel module for your device listed, e.g.:

qat_c62x               5626  0
intel_qat              82336  1 qat_c62x

Next, you need to expose the Virtual Functions (VFs) using the sysfs file system.

First find the BDFs (Bus-Device-Function) of the physical functions (PFs) of your device, e.g.:

lspci -d:37c8

You should see output similar to:

1a:00.0 Co-processor: Intel Corporation Device 37c8
3d:00.0 Co-processor: Intel Corporation Device 37c8
3f:00.0 Co-processor: Intel Corporation Device 37c8

Enable the VFs for each PF by echoing the number of VFs per PF to the pci driver:

echo 16 > /sys/bus/pci/drivers/c6xx/0000:1a:00.0/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/0000:3d:00.0/sriov_numvfs
echo 16 > /sys/bus/pci/drivers/c6xx/0000:3f:00.0/sriov_numvfs

Check that the VFs are available for use. For example lspci -d:37c9 should list 48 VF devices available for a C62x device.

To complete the installation follow the instructions in Binding the available VFs to the vfio-pci driver.

---

#链接start

1.2. Binding the available VFs to the vfio-pci driver

Note:

• Please note that due to security issues, the usage of older DPDK igb_uio driver is not recommended. This document shows how to use the more secure vfio-pci driver.
• If QAT fails to bind to vfio-pci on Linux kernel 5.9+, please see the QATE-39220 and QATE-7495 issues in IDZ doc which details the constraint about trusted guests and add disable_denylist=1 to the vfio-pci params to use QAT. See also this patch description.

Unbind the VFs from the stock driver so they can be bound to the vfio-pci driver.

1.2.1. For an Intel® QuickAssist Technology DH895xCC device

The unbind command below assumes BDFs of 03:01.00-03:04.07, if your VFs are different adjust the unbind command below:

cd to the top-level DPDK directory
for device in $(seq 1 4); do \
    for fn in $(seq 0 7); do \
        usertools/dpdk-devbind.py -u 0000:03:0${device}.${fn}; \
    done; \
done

1.2.2. For an Intel® QuickAssist Technology C62x device

The unbind command below assumes BDFs of 1a:01.00-1a:02.07, 3d:01.00-3d:02.07 and 3f:01.00-3f:02.07, if your VFs are different adjust the unbind command below:

cd to the top-level DPDK directory
for device in $(seq 1 2); do \
    for fn in $(seq 0 7); do \
        usertools/dpdk-devbind.py -u 0000:1a:0${device}.${fn}; \
        usertools/dpdk-devbind.py -u 0000:3d:0${device}.${fn}; \
        usertools/dpdk-devbind.py -u 0000:3f:0${device}.${fn}; \
    done; \
done

1.2.3. For Intel® QuickAssist Technology C3xxx or 200xx or D15xx device

The unbind command below assumes BDFs of 01:01.00-01:02.07, if your VFs are different adjust the unbind command below:

cd to the top-level DPDK directory
for device in $(seq 1 2); do \
    for fn in $(seq 0 7); do \
        usertools/dpdk-devbind.py -u 0000:01:0${device}.${fn}; \
    done; \
done

1.2.4. Bind to the vfio-pci driver

Load the vfio-pci driver, bind the VF PCI Device id to it using the dpdk-devbind.py script then use the --status option to confirm the VF devices are now in use by vfio-pci kernel driver, e.g. for the C62x device:

cd to the top-level DPDK directory
modprobe vfio-pci
usertools/dpdk-devbind.py -b vfio-pci 0000:03:01.1
usertools/dpdk-devbind.py --status

Use modprobe vfio-pci disable_denylist=1 from kernel 5.9 onwards. See note in the section Binding the available VFs to the vfio-pci driver above.

1.2.5. Testing

QAT SYM crypto PMD can be tested by running the test application:

cd ./<build_dir>/app/test
./dpdk-test -l1 -n1 -a <your qat bdf>
RTE>>cryptodev_qat_autotest

QAT ASYM crypto PMD can be tested by running the test application:

cd ./<build_dir>/app/test
./dpdk-test -l1 -n1 -a <your qat bdf>
RTE>>cryptodev_qat_asym_autotest

QAT compression PMD can be tested by running the test application:

cd ./<build_dir>/app/test
./dpdk-test -l1 -n1 -a <your qat bdf>
RTE>>compressdev_autotest

1.2.6. Debugging

There are 2 sets of trace available via the dynamic logging feature:

• pmd.qat.dp exposes trace on the data-path.
• pmd.qat.general exposes all other trace.

pmd.qat exposes both sets of traces. They can be enabled using the log-level option (where 8=maximum log level) on the process cmdline, e.g. using any of the following:

--log-level="pmd.qat.general,8"
--log-level="pmd.qat.dp,8"
--log-level="pmd.qat,8"

Note

The global RTE_LOG_DP_LEVEL overrides data-path trace so must be set to RTE_LOG_DEBUG to see all the trace. This variable is in config/rte_config.h for meson build. Also the dynamic global log level overrides both sets of trace, so e.g. no QAT trace would display in this case:

--log-level="7" --log-level="pmd.qat.general,8"

#链接end

---

Note

If the QAT kernel modules are not loaded and you see an error like Failed to load MMP firmware qat_895xcc_mmp.bin in kernel logs, this may be as a result of not using a distribution, but just updating the kernel directly.

Download firmware from the kernel firmware repo.

Copy qat binaries to /lib/firmware:

cp qat_895xcc.bin /lib/firmware
cp qat_895xcc_mmp.bin /lib/firmware

Change to your linux source root directory and start the qat kernel modules:

insmod ./drivers/crypto/qat/qat_common/intel_qat.ko
insmod ./drivers/crypto/qat/qat_dh895xcc/qat_dh895xcc.ko

Note

If you see the following warning in /var/log/messages it can be ignored: IOMMU should be enabled for SR-IOV to work correctly.

1.3. Installation using IDZ QAT driver

浏览器输入https://01.org/ 进入Projects->QuickAssist Technology

或：

Download the latest QuickAssist Technology Driver from Intel Developer Zone. Consult the Quick Start Guide at the same URL for further information.

The steps below assume you are:

• Building on a platform with one C62x device.
• Using package qat1.7.l.4.2.0-000xx.tar.gz.
• On Fedora26 kernel 4.11.11-300.fc26.x86_64.

In the BIOS ensure that SRIOV is enabled and VT-d is disabled.

Uninstall any existing QAT driver, for example by running:

• ./installer.sh uninstall in the directory where originally installed.

Build and install the SRIOV-enabled QAT driver:

mkdir /QAT
cd /QAT

# Copy the package to this location and unpack
tar zxof qat1.7.l.4.2.0-000xx.tar.gz

./configure --enable-icp-sriov=host
make install

You can use cat /sys/kernel/debug/qat<your device type and bdf>/version/fw to confirm the driver is correctly installed and is using firmware version 4.2.0. You can use lspci -d:37c9 to confirm the presence of the 16 VF devices available per C62x PF.

Confirm the driver is correctly installed and is using firmware version 4.2.0:

cat /sys/kernel/debug/qat<your device type and bdf>/version/fw

Confirm the presence of 48 VF devices - 16 per PF:

lspci -d:37c9

To complete the installation - follow instructions in 1.2.Binding the available VFs to the vfio-pci driver

Note

If using a later kernel and the build fails with an error relating to strict_stroul not being available apply the following patch:

/QAT/QAT1.6/quickassist/utilities/downloader/Target_CoreLibs/uclo/include/linux/uclo_platform.h
+ #if LINUX_VERSION_CODE >= KERNEL_VERSION(3,18,5)
+ #define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (kstrtoul((str), (base), (num))) printk("Error strtoull convert %s\n", str); }
+ #else
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,38)
#define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; if (strict_strtoull((str), (base), (num))) printk("Error strtoull convert %s\n", str); }
#else
#if LINUX_VERSION_CODE >= KERNEL_VERSION(2,6,25)
#define STR_TO_64(str, base, num, endPtr) {endPtr=NULL; strict_strtoll((str), (base), (num));}
#else
#define STR_TO_64(str, base, num, endPtr)                                 \
     do {                                                               \
           if (str[0] == '-')                                           \
           {                                                            \
                *(num) = -(simple_strtoull((str+1), &(endPtr), (base))); \
           }else {                                                      \
                *(num) = simple_strtoull((str), &(endPtr), (base));      \
           }                                                            \
     } while(0)
+ #endif
#endif
#endif

Note

If the build fails due to missing header files you may need to do following:

sudo yum install zlib-devel
sudo yum install openssl-devel
sudo yum install libudev-devel

Note

If the build or install fails due to mismatching kernel sources you may need to do the following:

sudo yum install kernel-headers-`uname -r`
sudo yum install kernel-src-`uname -r`
sudo yum install kernel-devel-`uname -r`

Note

If the build fails on newer GCC versions (such as GCC 12) with an error relating to -lc not being found, apply the following patch:

/QAT/quickassist/lookaside/access_layer/src/Makefile
cd $(ICP_FINAL_OUTPUT_DIR);\
cmd="$(LINKER) $(LIB_SHARED_FLAGS) -o \
  $(LIB_SHARED) $(ADDITIONAL_OBJECTS) $(ADDITIONAL_LIBS) *.o -lpthread -ludev \
- -Bstatic -L$(ADF_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL) \
- -ladf_user -L$(OSAL_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL)/ \
- -losal -Bdynamic -lc"; \
+ -Bstatic -L$(ADF_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL) \
+ -ladf_user -L$(OSAL_DIR)/src/build/$(ICP_OS)/$(ICP_OS_LEVEL)/ \
+ -losal -Bdynamic -L/lib/x86_64-linux-gnu/ -lc"; \
echo "$$cmd"; \
$$cmd

Followed by this patch:

/QAT/quickassist/build_system/build_files/OS/linux_common_user_space_rules.mk
@echo 'Creating shared library ${LIB_SHARED}'; \
cd $($(PROG_ACY)_FINAL_OUTPUT_DIR);\
-  echo $(LINKER) $(LIB_SHARED_FLAGS) -o $@  $(OBJECTS) $(ADDITIONAL_OBJECTS) -lc;\
-  $(LINKER) $(LIB_SHARED_FLAGS)