GPU Passthrough is all the rage. The ArchLinux Wiki does an amazing job explaining all the different ways to setup VFIO/GPU passthrough. These are my notes for setting up GPU Passthrough in Fedora 30 with two identical cards. This does not aim to be a comprehensive guide.

I believe, while untested, this method can be used for setups with a different cards as well.

Setup KVM

Install KVM

sudo dnf -y install bridge-utils libvirt virt-install qemu-kvm virt-top libguestfs-tools virt-manager

Configure bridge networking using NetworkManager (this is largely due to laziness when setting the OS up, NM is the default, so I kinda just deal with it…)

# make up a bridge name, I like br0
export BR_NAME=br0
# this will vary depending on the mobo/eth card.  see `ip addr` to find device name
export BR_INTERFACE=enp5s0

# Get our device UUID
BR_INT_OG_UUID=$(nmcli -g GENERAL.CON-UUID device show "${BR_INTERFACE}")

# Create bridge
nmcli connection add type bridge autoconnect yes con-name "{BR_NAME}" ifname "${BR_NAME}"

# Disable STP
nmcli connection modify "${BR_NAME}" bridge.stp no

# Add our interface to our bridge
nmcli connection add type bridge-slave autoconnect yes con-name ${BR_INTERFACE} ifname ${BR_INTERFACE} master ${BR_NAME}

# Turn off our old interface
nmcli con down "${BR_INTERFACE}"

# Turn on our new bridge
nmcli con up "${BR_NAME}"

# delete our old interface config
nmcli con delete "${BR_INT_OG_UUID}"

Update Kernel CLI parameters

Append to GRUB_CMDLINE_LINUX in /etc/default/grub to enable iommu and load our vfio-pci driver. We do this early to ensure vfio is able to claim the devices before the kernel does.

GRUB_CMDLINE_LINUX="amd_iommu=on iommu=pt rd.driver.pre=vfio-pci"

Modprobe Config

Create /etc/modprobe.d/vfio.conf to tell modprobe how to load vfio-pci

install vfio-pci /sbin/vfio-pci-override.sh

Dracut Config

Create /etc/dracut.conf.d/vfio.conf to load our drivers and setup script into our initramfs.

add_drivers+="vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
install_items+="/sbin/vfio-pci-override.sh /usr/bin/dirname"

VGA Passthrough Script

Create /sbin/vfio-pci-override.sh, this will passthrough all non-boot VGA devices, including audio and USB controllers if they exist.

Update our boot system

# update initramfs
# or dracut -f --regenerate-all
dracut -f --kver $(uname -r)

# update grub
grub2-mkconfig > /etc/grub2-efi.cfg

Post setup Check

(this could be improved)

“failed to setup container for group XX: failed to set iommu for container: Operation not permitted”

Shoutout to /u/degerdem and /u/WiFivomFranman for identifying that you need to go into the BIOS

  • Advanced -> AMD PBS ** Enumerate all IOMMU in IVRS = Enabled

Nvidia GeForce Card reports Error 43 in Guest OS

In the <features></features> stanza, add ` to the hyperv block and hidden state=on`.

Before:

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <vmport state='off'/>
  </features>

After:

  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='10DE'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>