GPU passthrought setup for Nvidia V100 (Part I)

This is an instruction based on V100 and GPU compute purpose only. There will be two parts for this instruction, Host setup and Guest Setup 

Part one: HOST

Part two: Guest

Please make sure using Nvidia Tesla production, which means Maxwell,

Pascal, and Volta. We do not have hardware matrix from Nvidia yet.

Please also make sure you have an extra display card on the host at the meantime, or a SSH enviroment at least.

 

1. HOST enviroment verification

1.1 Make sure Your HOST is SLES12SP3 and so on

baird:~/:[0]# cat /etc/issue

Welcome to SUSE Linux Enterprise Server 15  (x86_64) - Kernel \r (\l).

1.2  Make sure your HOST support VT-d and being enabled from BIOS:

baird:~/:[0]# dmesg | grep -e "Directed I/O"

[   12.819760] DMAR: Intel(R) Virtualization Technology for Directed I/O

1.3 Make sure if you an extra GPU or VGA card:

baird:~/:[0]# lspci | grep -i "vga"

07:00.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA

G200e [Pilot] ServerEngines (SEP1) (rev 05)

baird:~/:[0]# lspci | grep -i nvidia

03:00.0 3D controller: NVIDIA Corporation GV100 [Tesla V100 PCIe] (rev a1)

 

2. Enable IOMMU

vim /etc/default/grub

# Make this line look like this

GRUB_CMDLINE_LINUX="intel_iommu=on iommu=pt rd.driver.pre=vfio-pci"

grub2-mkconfig -o /boot/grub2/grub.cfg

After reboot, you could verify by

dmesg |  grep -e DMAR -e IOMMU

 

3. Add nouveau to blacklist

baird:~/:[0]# vim /etc/modprobe.d/50-blacklist.conf

add "blacklist nouveau"

 

4. Setup VFIO and isolate the GPU used for pass-through

Add a file under /etc/modprobe.d

baird:~/:[0]# cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1db4

10de:1db4 is vender id and model id, lspci -nn will give you these values

baird:~/:[0]# lspci -nn | grep 03:00.0

03:00.0 3D controller [0302]: NVIDIA Corporation GV100 [Tesla V100 PCIe]

[10de:1db4] (rev a1)

5. load VFIO driver

baird:~/:[0]# modprobe vfio-pci

or add to your initrd file

baird:~/:[0]# cat /etc/dracut.conf.d/gpu-passthrough.conf

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"

dracut --force /boot/initrd $(uname -r)

 

6. Reboot Host and check GPU is isolated in different iommu group and

vfio driver is in use

find /sys/kernel/iommu_groups/*/devices/*

/sys/kernel/iommu_groups/47/devices/0000:03:00.0

/sys/kernel/iommu_groups/49/devices/0000:07:00.0

lspci -k

03:00.0 3D controller: NVIDIA Corporation GV100 [Tesla V100 PCIe] (rev a1)

        Subsystem: NVIDIA Corporation Device 1214

        Kernel driver in use: vfio-pci

        Kernel modules: nouveau