Accelerate your VM by OVS-DPDK in SUSE(HOST)

This article shows how to setup a vhost port in your vm with virtio device, implement a zero copy path for your virtual machines. 

Host Setup

1. Backgroud

2. Pre-requirement

3. Host Setup

3.1 cpu pin

3.2 numa

3.3 huge page

3.4 vfio

4. OVS+DPDK

4.1 dpdk

4.2 ovs

Guest Setup

4.3 vm config

5 VM setup

5.1 QEMU usuage

5.2 Libvirt usuage

6. Common issues

7. Reference

 

1. Background

dpdk, which is data plane development kit, it is a set of libraries to accelerate packet processing by running in userspace level, it is also used a bunch of other optimizations, like cpu affinaity, numa, huge page and so on.

This document is more of a tutorial that should guide you through all of the steps of installing DPDK and Open vSwitch from the packages, and then setup vhost-user ports  for a vm running by qemu command line or libvirt, it will also record  some comman issues.

 

2. Pre-requirement 

There are a couple things need to know before running dpdk

2.1, make sure your nic supports dpdk first 

https://core.dpdk.org/supported/

2.2

cpu pin, numa, and huge page support

 

2.3 software requirment

kernel > 3.2

glibc > 2.7

3. Host Setup

3.1 cpu core mask

3.2. numa

This is not necessary, uma could also work

We want to make sure to run the VM on the same NUMA node as ovs-vswitchd and as the backing NIC.

To determine which NUMA node a PCI device (NIC) is on, you can cat /sys/class/net/eth<#>/device/numa_node to see either a 0 or 1.

3.3. huge page setup

3.3.1 grub command line

default_hugepagesz=1G hugepagesz=1G hugepages=4

grub2-mkconfig -o /boot/grub2/grub.cfg

3.3.2 dynamic

echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages

On a NUMA machine, pages should be allocated explicitly on separate nodes:

echo 1024 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages

echo 1024 > /sys/devices/system/node/node1/hugepages/hugepages-2048kB/nr_hugepages

# echo 4 > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages

# echo 1024 > /sys/devices/system/node/node3/hugepages/hugepages-2048kB/nr_hugepages

3.3.3. mount huge pages before use

mkdir /mnt/huge

mount -t hugetlbfs nodev /mnt/huge

# mkdir /dev/hugepages1G

# mount -t hugetlbfs -o pagesize=1G none /dev/hugepages1G

but since I did not bind the right dpdk driver,

# mkdir /dev/hugepages2M

# mount -t hugetlbfs -o pagesize=2M none /dev/hugepages2M

it could also be made permanet by /etc/fstab

 nodev /mnt/huge hugetlbfs defaults 0 0

for 1G page_size

nodev /mnt/huge_1GB hugetlbfs pagesize=1GB 0 0

make sure restart libvirtd

3.4 VFIO

make sure your cpu supports vt-d and iommu first.

VFIO is preferred for latest DPDK, because it has better performance than UIO

Once your hardware supports it, just add below in grub command line

 

4. OVS+DPDK

DPDK will work as a library for OVS, current SLES disable it, however opensuse included dpdk for ovs.

Will discuss with Network team further for the reason why SLE disabled it.

 

For verify, you could run below command:

linux-3txe:~ # ovs-vswitchd --version

ovs-vswitchd (Open vSwitch) 2.10.1

DPDK 18.02.2    

4.1.1 DPDK setup

bind device, make sure the module is already loaded in kernel

modprobe igb_uio

dpdk_devbind --status

dpdk_devbind --bind=igb_uio 0000:02:00.0

dpdk_devbind --unbind 0000:02:00.0

For vfio-pci

modprobe vfio

modprobe vfio_pci

Sometime, need to bind special driver from vender directly.

4.2 OVS setup

For OVS that does not include dpdk, 

need to build from source

For thoses includes DPDK

ovs-vsctl --no-wait init

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-lcore-mask=0x6

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-socket-mem=1024

ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true