Install Debian VM (part 3 of Debian as type 1 hypervisor)

09 Aug 2020
#Debian 10#Virtualization#Linux

To make the Debian host OS acting like a type 1 hypervisor, the desktop OS must run as a virtual machine too. So the second GPU will be passthrough to the second virtual machine. This will leave the host without GPU, so make sure you can access the host through SSH.

When this machine works as intended you will get back to the host console everytime you shutdown this new virtual machine.

Lets get begin with the installation of Debian in a second virtual machine. Start with creating a folder to store files related to the virtual machine

# mkdir -p /kvm/debian

Download an install ISO of choice, it should be the same regardless of Linux distribution you choose.

Create a root drive and a home drive

qemu-img create -f qcow2 /kvm/debian/root.qcow2 20G
qemu-img create -f qcow2 /kvm/debian/home.qcow2 50G

I like to keep them on separate virtual disks, this way I can use the same /home for multiple Linux distros, as long as I don’t run them at the same time.

# virt-install \
   --name=debian \
   --cpu host-passthrough --hvm --features kvm_hidden=on \
   --ram=6144 \
   --vcpus=4 \
   --os-type=linux \
   --boot uefi --machine q35 \
   --disk path=/kvm/debian/root.qcow2,bus=virtio \
   --disk path=/kvm/debian/home.qcow2,bus=virtio \
   --cdrom /kvm/iso/debian.iso \
   --graphics vnc,listen=0.0.0.0 --noautoconsole

After install dump the XML file, using virsh dumpxml debian > /kvm/debian/debian.xml

And again we uncomment the XML node for the install ISO, because Linux already has virtio drivers we only need to care about the cdrom.

<!--
<disk type='file' device='cdrom'>
  <driver name='qemu' type='raw'/>
  <target dev='sdc' bus='sata'/>
  <source file='/kvm/iso/debian.iso' />
  <readonly/>
  <address type='drive' controller='0' bus='0' target='0' unit='2'/>
</disk>
-->

and we redefine the virtual machine with virsh define /kvm/debian/debian.xml command.

Passing through GPU

We already did a lot of this on our Windows VM installation, so I wont cover everything again.

Find GPU id

Lets use lspci again to find our second GPU, in my case I want the GTX1050ti and this is the row I am looking for

25:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)

and running lspci -s 25:00 only lists two subdevices for this card

25:00.0 VGA compatible controller: NVIDIA Corporation GP107 [GeForce GTX 1050 Ti] (rev a1)
25:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)

So in this case we only need to add two hostdev nodes into our XML file, in the <devices> section

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x25' slot='0x00' function='0x0'/>
  </source>
  <rom file='/kvm/debian/GTX1050Ti.rom' />
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
    <address domain='0x0000' bus='0x25' slot='0x00' function='0x1'/>
  </source>
</hostdev>

As you can see, this time we have a <rom file='path-to-rom-file' /> block in our XML code. This rom file is a must to have when you share the GPU between the host and guest.

Get ROM file for our GPU

This is the hard part, to find a working ROM file. I have found lot’s of guides for this. One way is to extract the ROM file using GPU-Z, this method requires you to have a bare metal Windows install because GPU-Z is for Windows and to access the GPU ROM without anything tainting the ROM it needs to run as normal install and not in a VM.

There is also several places to download ROM files for different GPU’s. But remember that the ROM file must match the ROM file on your card, so the ROM file has to be the same GPU, manufacturer and version as your card have.

Once again the Arch WIKI guide is a good source for information about extracting ROM files.

Download the rom-parser tool and compile it

# git clone https://github.com/awilliam/rom-parser
# cd rom-parser && make

Extract the ROM to a file, replace 25:00.0 with your VGA card identifier from the lspci output earlier.

# echo 1 > /sys/bus/pci/devices/0000:25:00.0/rom
# cat /sys/bus/pci/devices/0000:25:00.0/rom > ./GTX1050Ti.rom
# echo 0 > /sys/bus/pci/devices/0000:25:00.0/rom

Use the rom-parser tool to verify the rom image file

# ./rom-parser GTX1050Ti.rom
Valid ROM signature found @a00h, PCIR offset 1a0h
        PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1c82, class: 030000
        PCIR: revision 0, vendor revision: 1
Valid ROM signature found @fa00h, PCIR offset 1ch
        PCIR: type 3 (EFI), vendor: 10de, device: 1c82, class: 030000
        PCIR: revision 3, vendor revision: 0
                EFI: Signature Valid, Subsystem: Boot, Machine: X64
        Last image

To be compatible with UEFI (which we are using) you must have the “type 3 (EFI)” ROM in the rom file. You may need to update your VBIOS from your manufacturer if you don’t have any “type 3 (EFI)” ROM part in your GPU ROM. But I think most modern cards already have this.

ACS Override patch the kernel

So in the previus part, when we installed Windows I mentioned iommu groups and attached a short script (again from Arch WIKI). Using this script showed me that my second GPU, the GTX1050Ti, is not in it’s own iommu group. So it can’t be passed through to a virtual machine, it maybe works if you passthrough all the devices of that group. But that is not always possible, it may be in the same group as the SATA controller you are using to boot your host with.

But there is a solution the ACS Overide patch which is an dirty hack to the linux kernel to separate devices into more iommu groups. This patch is considered a security risk if I remember correctly, thats why it’s not already in the kernel.

So I made a bash script to build the kernel for me, makes my life easier next time I need to patch and build the kernel (and it will be easier if I want to re-install my host)

#!/bin/bash
###########################################################
#
# Build custom Debian kernel with ACS Override patch
#
# Build instructions from
# https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official
#
# ACSO: https://queuecumber.gitlab.io/linux-acs-override/
#
set +x

CWD=$(pwd)

apt-get install linux-source-5.4
cd /usr/src && tar xaf linux-source-5.4.tar.xz
cd /usr/src/linux-source-5.4

patch -p1 < $CWD/acso.patch

make -j12 deb-pkg LOCALVERSION=-acs

dpkg -i /usr/src/linux-image-5.4.19-acs_5.4.19-acs-1_amd64.deb

If you have more or less than 6 cores 12 threads like I have, you can change the line make -jNN deb-pkg LOCALVERSION=-acs so that NN is the number of threads your CPU have.

Now we need to tell the kernel to use the ACS Overide patch, this is done through the Grub command line, so we need to change /etc/default/grub again and add pcie_acs_override=downstream,multifunction to the command line. Should look something like this now

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction"

After that we need to run update-grub and reboot our machine. After reboot we can run the script from Arch WIKI again

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

This script is from Arch WIKI - PCI passthrough via OVMF. To make sure our graphics card is in a separate iommu group.

One last thing about GPU sharing between host and guest

From my experience while building my setup, I get this error message when I boot my host machine and when I shut down my virtual machine.

nouveau 0000:25:00.0: DRM: failed to create kernel channel -22

When I found the solution and solved the error, it no longer works. I no longer get my console back when I shutdown the virtual machine.

I fail to remember what I did to solve this problem, I only remember I had to revert the solution for the above error message.

Passthrough sound card

So to pass through the sound card it is the same process as the graphics card, lets use lspci again to find our sound card. The row for my sound card looks like this

28:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

it has the identifier 28:00.4 so this only need one <hostdev> node to add.

<hostdev mode='subsystem' type='pci' managed='yes'>
  <source>
   <address domain='0x0000' bus='0x28' slot='0x00' function='0x4'/>
  </source>
</hostdev>

FLR Patch the kernel

Apperantly the AMD Starship/Matisse HD soundcard does not work properly when it is passed through to a virtual machine.

So I need to patch the kernel again with the FLR patch.

Time to update my kernel-patching script and add the FLR patch

#!/bin/bash
###########################################################
#
# Build custom Debian kernel with ACS Override patch
# and AMD Ryzen 3000 FLR patch for passthrough bug on
# Audio device etc.
#
# Build instructions from
# https://kernel-team.pages.debian.net/kernel-handbook/ch-common-tasks.html#s-common-official
#
# ACSO: https://queuecumber.gitlab.io/linux-acs-override/
# FLR: https://www.reddit.com/r/VFIO/comments/eba5mh/workaround_patch_for_passing_through_usb_and/
#
set +x

CWD=$(pwd)

apt-get install linux-source-5.4
cd /usr/src && tar xaf linux-source-5.4.tar.xz
cd /usr/src/linux-source-5.4

patch -p1 < $CWD/acso.patch
patch -p1 < $CWD/flr.patch

make -j12 deb-pkg LOCALVERSION=-acs-flr

dpkg -i /usr/src/linux-image-5.4.19-acs-flr_5.4.19-acs-flr-1_amd64.deb

Now we need to tell the kernel to use the FLR patch, this is done through the Grub command line, so we need to change /etc/default/grub again and add pcie_no_flr=1022:1487 to the command line. The 1022:1487 identifier string is found using the lspci command like this

# lspci -s 28:00.4 -nn
28:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

and at the end of the string inside brackets is our id string to append in the kernel command line.

Kernel command line should look something like this now

GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on pcie_acs_override=downstream,multifunction pcie_no_flr=1022:1487"

After that we need to run update-grub and reboot our host machine.

USB Passthroug for keyboard and mouse

Passthrough USB mouse and keyboard is the same for this Linux virtual machine as it was for my previous Windows virtual machine.

But this time I wrote a script for the process, it may be useful later

#!/bin/bash
# usb-attach.sh

# Abort script execution on errors
set -e

BUS=$1
DEVICE=$2
DOMAIN=$3

function howto {
  echo "Usage: $0 <Bus> <Device> <Domain>"
  echo
  echo "Use lsusb to get Bus and Device numbers"
  echo
  exit
}

if [ -z "${BUS}" ]; then
  echo "Missing <Bus> parameter."
  howto
fi
if [ -z "${DEVICE}" ]; then
  echo "Missing <Device> parameter."
  howto
fi
if [ -z "${DOMAIN}" ]; then
  echo "Missing <Domain> parameter."
  howto
fi
    
# make sure numbers has no leading zeros, else libvirt will freak out
BUS=$((10#$BUS))
DEVICE=$((10#$DEVICE))

# Use virsh to attach device to VM
echo "Running virsh attach-device ${DOMAIN} for USB bus=${BUS} device=${DEVICE}:"
virsh "attach-device" "${DOMAIN}" /dev/stdin <<END
<hostdev mode='subsystem' type='usb'>
  <source>
    <address bus='${BUS}' device='${DEVICE}' />
  </source>
</hostdev>
END

So using this script I just have to run lsusb to find what USB device to passthrough

lsusb
Bus 006 Device 002: ID 2109:0812 VIA Labs, Inc. VL812 Hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 002: ID 2109:2812 VIA Labs, Inc. VL812 Hub
Bus 005 Device 005: ID 03f0:0024 HP, Inc KU-0316 Keyboard
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 046d:c051 Logitech, Inc. G3 (MX518) Optical Mouse
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

So to passthrough HP keyboard and Logitech mouse I just have to run my script

./usb-attach.sh 5 5 debian
Running virsh attach-device debian for USB bus=5 device=5:
Device attached successfully

./usb-attach.sh 1 2 debian
Running virsh attach-device debian for USB bus=1 device=2:
Device attached successfully

Just a warning though, using this script will attach the USB device for the current session. When the virtual machine is powered off it will forget those devices.

To make it permanent you have to add the <hostdev> block to the virtual machine XML file as before

<hostdev mode='subsystem' type='usb'>
  <source>
    <address bus='5' device='5' />
  </source>
</hostdev>
<hostdev mode='subsystem' type='usb'>
  <source>
    <address bus='1' device='2' />
  </source>
</hostdev>

And make sure libvirt updates the virtual machine

virsh define /kvm/debian/debian.xml

Thats it for now.

Raddinox © 2024 - Powered by