I have a brand new Gigabyte x570 Aorus Pro, this motherboard has super awesome iommu groups. Finally can I use both my NVME drives and two GPUs. My old Ryzen 5 3600 cpu died, but I got my hands on a pre-owned Ryzen 9 3950x. It exploded my budget so I'm still stuck on 16Gb ram. But now I have more than twice as much cores, great for playing around with virtualization.
I'm not going full type-1 hypervisor in this guide. I will be running my latest Debian SID with Xanmod-cacule kernel install, with a side-loaded VM (or what to call it). I will passthrough my Nvidia GTX1050Ti and I will be running a second keyboard, monitor and mouse setup for that machine. To run it as if it was another desktop machine.
I am running the Xanmod-cacule kernel, and according to the
/boot/config-5.14.15-xanmod1-cacule, VFIO is built into the kernel. So there is no need to load any kernel modules.
You can check it like this
$ cat /boot/config-5.14.15-xanmod1-cacule | grep VFIO CONFIG_KVM_VFIO=y CONFIG_VFIO_IOMMU_TYPE1=y CONFIG_VFIO_VIRQFD=y CONFIG_VFIO=y CONFIG_VFIO_NOIOMMU=y CONFIG_VFIO_PCI=y CONFIG_VFIO_PCI_VGA=y CONFIG_VFIO_PCI_MMAP=y CONFIG_VFIO_PCI_INTX=y CONFIG_VFIO_PCI_IGD=y CONFIG_VFIO_MDEV=m # CONFIG_SAMPLE_VFIO_MDEV_MTTY is not set # CONFIG_SAMPLE_VFIO_MDEV_MDPY is not set # CONFIG_SAMPLE_VFIO_MDEV_MDPY_FB is not set # CONFIG_SAMPLE_VFIO_MDEV_MBOCHS is not set
Everything with an
y is built into the kernel, if it is an
m it is built as a module and can be loaded. The only thing with an
CONFIG_VFIO_MDEV and we are not going to use that one.
Anyway, if you need to load the VFIO kernel modules at boot. Just create a config file so the kernel module loader knows what to load.
# /etc/modules-load.d/vfio.conf # Load VFIO kernel modules vfio vfio_iommu_type1 vfio_pci vrio_irqfd
To find the Device ID for our GPU we will use
$ lspci ... 0a:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1) ...
But this isn't all of it, my 1050ti has and audio device as well, actuall all the 0a:00.x devices has to be passed through to the VM. To list all of them including the device ID we are looking for type
lspci -s 0a:00 -nn
$ lspci -s 0a:00 -nn 0a:00.0 VGA compatible controller : NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] [10de:1c82] (rev a1) 0a:00.1 Audio device : NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
As you can see, I have two devices I need to passthrough. So I need to remember, or write down, the two long ID's in brackets. For me it's
To make sure the VFIO-PCI driver takes control of the Nvidia card before the Nvidia drivers I had to add some kernel command line arguments.
I have added
/etc/default/grub. This will make sure that vfio driver will load for that particular device ID.
I don't know if it is needed, but I have created a soft dependency between the Nvidia and VFIO-PCI driver to make sure the VFIO driver loads first.
Create the file
# /etc/modprobe.d/vfio-nvidia.conf # Try to load VFIO driver before Nvidia softdep nvidia pre: vfio-pci
This will trick the kernel module loader to think the Nvidia driver has some dependency on the vfio driver, so it will load the vfio driver first.
To enable IOMMU so we can use the passthrough feature. This is enabled using a kernel command line parameter in the Grub config.
I'm running AMD so I have to add
amd_iommu=on to my
So now my
GRUB_CMDLINE_LINUX_DEFAULT looks like this
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on vfio-pci.ids=10de:1c82,10de:0fb9"
For Intel users it will be
sudo update-grub to update the initramfs stuff.
Make sure you have a virtual machine using UEFI and not BIOS mode. Then we can add this XML block to the virtual machine's XML
<hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0' multifunction='on'/> </source> <rom bar='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/> </source> </hostdev>
Just remember to change the bus,slot and function address to your device address found using
To update my FreeBSD machine with the above GPU passthrough I first dump the XML from libvirt
$ virsh dumpxml freebsd > freebsd.xml
I usually dump my XML files for each virtual machine and save them. This way I still have the VM config even if I re-install my Debian host.
To re-define my virtual machine in libvirt, we need to run virsh again
$ virsh define freebsd.xml
This will update the already existing FreeBSD virtual machine because the XML file has an UUID string to identify each machine with.