Skip to content

1. Setting everything up

Jump to bottom
HikariKnight edited this page Dec 22, 2021 · 16 revisions

Setting everything up.

  1. Make sure virtualization is enabled in your UEFI/BIOS, it might be named VT-d or AMD-v or SVM, this is needed to literally even get started with anything here.

  2. Make sure that IOMMU is enabled in your UEFI/BIOS.

  3. Your OS must be installed in UEFI mode too.

  4. Only desktop computers can use VFIO easily, laptops can not use GPU Passthrough with VFIO reliably (really depends on how the laptop is assembled) and should not be attempted unless you know what you are doing.

Installing QEMU and Virt-Manager

  1. Open your terminal and install all the needed packages

    Note: Keep the terminal open for the remainder of the guide.
sudo apt install virt-manager qemu-kvm ovmf

Local “host-only” connection between Host and Guest

NOTE: this will be needed for any communication between the host and guest over the network, and you will need it! (and this is the easiest method to setup that)

  1. Make a template file for the isolated network.
nano /tmp/isolated.xml
  1. Paste in the configuration below (edit it to avoid using the same address range as your network! The configuration provided should not clash with any home network from my experience).
<network>
  <name>isolated</name>
  <ip address='10.10.10.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='10.10.10.2' end='10.10.10.254' />
    </dhcp>
  </ip>
</network>
  1. Save the file with CTRL+X and then run the below commands to import the network config and enable it for use in the Virtual Machine Manager.
virsh net-define /tmp/isolated.xml
virsh net-autostart isolated
virsh net-start isolated
  1. You will now be able to add the isolated virtual network to any new virtual network card, for the best performance you should set the device model to VirtIO.

  1. You will now be able to set up a file sharing to share files between the Host and the Guest should you wish to do that, you will also be able to set up the software based kvm switch mentioned later in this document. Host can be reached at 10.10.10.1 if you used the default config.

  2. Should you need to edit the isolated network settings in the future, use the below command.
virsh net-edit isolated

Enable IOMMU on your system (GRUB)

  1. Edit “/etc/default/grub” and tell the kernel to enable IOMMU at boot Note: there are different parameters for intel and amd!
sudo nano /etc/default/grub

  1. Add the below parameters based on your CPU manufacturer at the end of the GRUB_CMDLINE_LINUX_DEFAULT= but make sure they are inside the quotes so it looks something similar to GRUB_CMDLINE_LINUX_DEFAULT="quiet splash iommu=pt intel_iommu=on"

    Note: iommu=pt lets you passthrough pci devices without a performance penalty on the host, if you have problems passing through devices with iommu=pt then try iommu=1

AMD INTEL
iommu=pt amd_iommu=on iommu=pt intel_iommu=on

  1. Press CTRL+X then Enter/Return to save.

  2. Update the grub config with the below command.
sudo update-grub
  1. Reboot your system.

Enable IOMMU on your system (SystemD-Boot)

I will assume you have installed kernelstub and have configured it properly (this is done by default on Pop_OS!)

  1. Run this command to add the parameters to systemd-boot
AMD INTEL
sudo kernelstub -a "iommu=pt amd_iommu=on" sudo kernelstub -a "iommu=pt intel_iommu=on"

Make a ls-iommu script just to make your life easier

  1. Create the file /usr/local/bin/ls-iommu with.
sudo nano /usr/local/bin/ls-iommu
  1. Paste the below bash script into the file and save with CTRL+X

Note: Code gotten from here originally by Wendell from Level1Techs.

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done
  1. Now make the script executable with
sudo chmod +x /usr/local/bin/ls-iommu
  1. The script is now executable and you will be able to run it with.
ls-iommu

  1. The output should look something like this. ls-iommu output

  2. Make sure that the VGA device you want to passthrough is in its own group along with its audio device, like in this example for my GTX 1070 ti. ls-iommu example

If the VGA controller and accompanying audio device are not in their own group together with nothing else in the group, you will have to apply the ACS override patch to the kernel, which is outside the scope of this guide as I have not personally needed it yet. If anyone knows how to apply it then message me on twitter and I will gladly include it! https://twitter.com/TheHikariKnight

Enabling vfio in initramfs

  1. Enable the vfio modules in initramfs by editing this file
sudo nano /etc/initramfs-tools/modules
  1. Append/add these lines if they do not already exist in there
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
  1. Press CTRL+X and save the file then run the below command to update the initramfs
sudo update-initramfs -u

Isolating/stubbing your GPU for passthrough

Warning: Before continuing with this, make sure you have a second, different model of GPU available, this can be an internal GPU like an Intel GPU on your CPU or a second dedicated Graphics Card (that is a different model! Or you might run into issues from my experience), make sure this GPU is used as your current Linux Graphics output! As the instructions below will make the stubbed GPU not usable in your Linux system until you undo this!

To guarantee that your GPU gets properly stubbed, we will stub it using modpobe, modules and the bootloader, this is important as different kernels have different loading priorities for VFIO options and adding the options in all locations will guarantee that the GPU gets stubbed before any driver loads.

  1. Depending on the card you want to pass through being Nvidia or AMD please create the modprobe config file for that gpu.
Nvidia GPU Passthrough AMD GPU Passthrough
sudo nano /etc/modprobe.d/nvidia.conf sudo nano /etc/modprobe.d/amdgpu.conf
  1. Paste in the below content based on your card to make sure vfio is loaded before your graphic card driver, then save the file with CTRL+X.
Nvidia GPU Passthrough AMD GPU Passthrough
softdep nvidia pre: vfio vfio_pci
softdep nouveau pre: vfio vfio_pci 		softdep amdgpu pre: vfio vfio_pci
  1. Now we need to find the PCI-e ID for the graphic card (and its audio controller) that we will be passing through to the virtual machine. To find it we will use the ls-iommu script we made earlier.
ls-iommu | grep -iP "VGA|audio"

  1. This should list up any VGA and audio devices, for my example I will be passing through my Nvidia GTX 1070 ti, I have highlighted the PCI-e ID in the output so you know what you are looking for (make sure you grab the ID for both the VGA device and the Audio device that belongs to the graphic card, you will need both!!). ls-iommu gpu

    If it is not noticable in the image, I am interested in the IDs which are inside the brackets at the end of the lines with the Nvidia device, these being 10de:1b82 and 10de:10f0 for me.
    If your card has extra devices like a usb and serial bus controller then you note down those IDs too for your passthrough.

  2. Make a modprobe config file for vfio.
sudo nano /etc/modprobe.d/vfio_pci.conf
  1. Paste the following into the file (replace my PCI-e IDs with your own!) Then save the file with CTRL+X.
options vfio_pci ids=10de:1b82,10de:10f0
  1. Edit /etc/modules
sudo nano /etc/modules
  1. Add this to the bottom of the file (replace 10de:1b82,10de:10f0 with your IDs) Save the file with CTRL+X
vfio
vfio_iommu_type1
vfio_pci ids=10de:1b82,10de:10f0
vfio_virqfd
  1. We now need to edit your bootloader to guarantee that the stubbing happens at all times.

Isolating/stubbing your GPU (using GRUB)

  1. Edit grub by editing /etc/default/grub
sudo nano /etc/default/grub
  1. Add vfio_pci.ids=10de:1b82,10de:10f0 to the end of the GRUB_CMDLINE_LINUX_DEFAULT line before the ", save by pressing CTRL+X and it should look something like below.
GRUB_CMDLINE_LINUX_DEFAULT="iommu=pt amd_iommu=on vfio_pci.ids=10de:1b82,10de:10f0"
  1. Update grub by running
sudo update-grub

Isolating/stubbing your GPU (using SystemD-Boot)

  1. For systemd-boot you I assume you have kernelstub installed and configured, you can add the kernel parameters by running
sudo kernelstub -a "vfio_pci.ids=10de:1b82,10de:10f0"

Verify that stubbing works

  1. Now that you have configured VFIO stubbing for your kernel and bootloader it is time to verify that it works.

  2. Reboot your computer then verify that the graphic card you stubbed is being used by the vfio-pci kernel driver using the below command.

    You are looking for the line “Kernel driver in use: vfio-pci” on the graphic card and its audio (and any other) device that you stubbed.
lspci -vnn | grep -iP "vga|amdgpu|nvidia|nouveau|vfio-pci"

Tell KVM to ignore MSRS to avoid a BSOD on newer Windows 10 builds (1803+)

  1. Make a new modprobe file for kvm.
sudo nano /etc/modprobe.d/kvm.conf
  1. Paste in the content from the box below.

    The first line tells kvm to ignore MSRS while the second line stops the kvm from spamming notifications into the tty that it has ignored MSRS.

options kvm ignore_msrs=1
options kvm report_ignored_msrs=0
  1. Save the file by pressing CTRL+X.

Edit apparmor permissions to give libvirt device access

Note: check if the folder /etc/apparmor.d/local/abstractions/ exists, if it does not exist then edit /etc/apparmor.d/abstractions/libvirt-qemu directly.

  1. Edit the abstractions file for libvirt-qemu.
sudo nano /etc/apparmor.d/local/abstractions/libvirt-qemu
  1. Add this into the file
 # allow usb passthrough
 /dev/bus/usb/** rw,
 /run/udev/data/* rw,
  1. Now go to the bottom of the file and add the permissions for looking-glass (which we will install later!).
 # Looking Glass
 /dev/shm/looking-glass rw,
  1. If you are planning to pass through a RAW block device (like an SSD) you can add them to the bottom of the file too, however instead of using /dev/sdX I will highly recommend you use the persistent names for the block devices instead, to avoid accidentally passing the wrong device if the letter changes after a reboot or change to the physical system (which can happen!).

    I personally use the WWID symbolic links for the block devices inside /dev/disk/by-id/ as this is guaranteed to be persistent and unique per storage device! The below box is an example based on me passing through my SSD. I would then use the same path I gave apparmor when I pass through that SSD, and it will guarantee that only that disk gets used for the virtual machine.

 # Raw Disk Access
 /dev/disk/by-id/ata-SAMSUNG_MZ7TD128HAFV-000L1_S14TNSAD626520 rw,
  1. Once done, save the file with CTRL+X and restart apparmor with the below command.
sudo service apparmor restart
Clone this wiki locally