Dual NVIDIA GPU with passthrough on Debian

01 Nov 2021
#Debian Sid#Virtualization#Libvirt#Linux

I have a brand new Gigabyte x570 Aorus Pro, this motherboard has super awesome iommu groups. Finally can I use both my NVME drives and two GPUs. My old Ryzen 5 3600 cpu died, but I got my hands on a pre-owned Ryzen 9 3950x. It exploded my budget so I’m still stuck on 16Gb ram. But now I have more than twice as much cores, great for playing around with virtualization.

I’m not going full type-1 hypervisor in this guide. I will be running my latest Debian Sid with Xanmod-cacule kernel install, with a side-loaded VM (or what to call it). I will passthrough my Nvidia GTX1050Ti and I will be running a second keyboard, monitor and mouse setup for that machine. To run it as if it was another desktop machine.

  1. Enable VFIO kernel modules
  2. Find the ID for our GPU
  3. Make sure VFIO drivers takes control of our GPU
  4. Enable IOMMU
  5. Edit virtual machine XML for GPU passthrough

Load VFIO modules

I am running the Xanmod-cacule kernel, and according to the /boot/config-5.14.15-xanmod1-cacule, VFIO is built into the kernel. So there is no need to load any kernel modules.

You can check it like this

$ cat /boot/config-5.14.15-xanmod1-cacule | grep VFIO
CONFIG_KVM_VFIO=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO_VIRQFD=y
CONFIG_VFIO=y
CONFIG_VFIO_NOIOMMU=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
CONFIG_VFIO_MDEV=m
# CONFIG_SAMPLE_VFIO_MDEV_MTTY is not set
# CONFIG_SAMPLE_VFIO_MDEV_MDPY is not set
# CONFIG_SAMPLE_VFIO_MDEV_MDPY_FB is not set
# CONFIG_SAMPLE_VFIO_MDEV_MBOCHS is not set

Everything with an y is built into the kernel, if it is an m it is built as a module and can be loaded. The only thing with an m is CONFIG_VFIO_MDEV and we are not going to use that one.

Anyway, if you need to load the VFIO kernel modules at boot. Just create a config file so the kernel module loader knows what to load.

edit /etc/modules-load.d/vfio.conf

# /etc/modules-load.d/vfio.conf
# Load VFIO kernel modules
vfio
vfio_iommu_type1
vfio_pci
vrio_irqfd

Find the Device ID’s for the GPU we want to passthrough

To find the Device ID for our GPU we will use lspci command

$ lspci
...
0a:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)
...

But this isn’t all of it, my 1050ti has and audio device as well, actuall all the 0a:00.x devices has to be passed through to the VM. To list all of them including the device ID we are looking for type lspci -s 0a:00 -nn

$ lspci -s 0a:00 -nn
0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1)
0a:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)

As you can see, I have two devices I need to passthrough. So I need to remember, or write down, the two long ID’s in brackets. For me it’s 10de:1c82 and 10de:0fb9

Make sure VFIO-PCI grabs the Nvidia card before Nvidia drivers

To make sure the VFIO-PCI driver takes control of the Nvidia card before the Nvidia drivers I had to add some kernel command line arguments.

I have added vfio-pci.ids=10de:1c82,10de:0fb9 to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub. This will make sure that vfio driver will load for that particular device ID.

I don’t know if it is needed, but I have created a soft dependency between the Nvidia and VFIO-PCI driver to make sure the VFIO driver loads first.

Create the file /etc/modprobe.d/vfio-nvidia.conf

# /etc/modprobe.d/vfio-nvidia.conf
# Try to load VFIO driver before Nvidia
softdep nvidia pre: vfio-pci

This will trick the kernel module loader to think the Nvidia driver has some dependency on the vfio driver, so it will load the vfio driver first.

Enable IOMMU

To enable IOMMU so we can use the passthrough feature. This is enabled using a kernel command line parameter in the Grub config.

I’m running AMD so I have to add amd_iommu=on to my GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub file

So now my GRUB_CMDLINE_LINUX_DEFAULT looks like this

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on vfio-pci.ids=10de:1c82,10de:0fb9"

For Intel users it will be intel_iommu=on

Run sudo update-grub to update the initramfs stuff.

Edit the virtual machine XML for GPU passthrough

Make sure you have a virtual machine using UEFI and not BIOS mode. Then we can add this XML block to the virtual machine’s XML

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0' multifunction='on'/>
  </source>
  <rom bar='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
  </source>
</hostdev>

Just remember to change the bus,slot and function address to your device address found using lspci

To update my FreeBSD machine with the above GPU passthrough I first dump the XML from libvirt

$ virsh dumpxml freebsd > freebsd.xml

I usually dump my XML files for each virtual machine and save them. This way I still have the VM config even if I re-install my Debian host.

To re-define my virtual machine in libvirt, we need to run virsh again

$ virsh define freebsd.xml

This will update the already existing FreeBSD virtual machine because the XML file has an UUID string to identify each machine with.

Raddinox © 2024 - Powered by