GitHub - QaidVoid/Complete-Single-GPU-Passthrough: Single GPU VFIO Passthrough Guide

Note: I'm currently not using this, so my ability to provide support is limited. If you encounter issues or have questions about the setup, I recommend asking at r/VFIO.

Disable Resizable BAR Support in BIOS settings. Cards that support Resizable BAR can cause problems with black screens following driver load if Resizable BAR is enabled in UEFI/BIOS. There doesn't seem to be a large performance penalty for disabling it, so turn it off for now until ReBAR support is available for KVM.

Set the kernel paramater depending on your CPU.

GRUB

Edit GRUB configuration

/etc/default/grub
`GRUB_CMDLINE_LINUX_DEFAULT="... intel_iommu=on iommu=pt ..."`
OR
`GRUB_CMDLINE_LINUX_DEFAULT="... amd_iommu=on iommu=pt ..."`

Generate grub.cfg

grub-mkconfig -o /boot/grub/grub.cfg

Systemd Boot

Edit boot entry.

/boot/loader/entries/*.conf
`options root=UUID=...intel_iommu=on iommu=pt..`
OR
`options root=UUID=...amd_iommu=on iommu=pt..`

Reboot your system for the changes to take effect.

To verify IOMMU, run the following command.

dmesg | grep IOMMU

The output should include the message Intel-IOMMU: enabled for Intel CPUs or AMD-Vi: AMD IOMMUv2 loaded and initialized for AMD CPUs.

To view the IOMMU groups and attached devices, run the following script:

#!/bin/bash
shopt -s nullglob
for g in `find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V`; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

When using passthrough, it is necessary to pass every device in the group that includes your GPU.
You can avoid having to pass everything by using ACS override patch.

Install required tools

Gentoo Linux

emerge -av qemu virt-manager libvirt ebtables dnsmasq

Arch Linux

pacman -S qemu libvirt edk2-ovmf virt-manager dnsmasq ebtables

Fedora

dnf install @virtualization

Ubuntu

apt install qemu-kvm qemu-utils libvirt-daemon-system libvirt-clients bridge-utils virt-manager ovmf

Enable required services

SystemD

systemctl enable --now libvirtd

OpenRC

rc-update add libvirtd default
rc-service libvirtd start

Sometimes, you might need to start default network manually.

virsh net-start default
virsh net-autostart default

Setup Guest OS

NOTE: You should replace win10 with your VM's name where applicable
You should add your user to libvirt group to be able to run VM without root. And, input and kvm group for passing input devices.

usermod -aG kvm,input,libvirt username

Download virtio driver.
Launch virt-manager and create a new virtual machine. Select Customize before install on Final Step.
In Overview section, set Chipset to Q35, and Firmware to UEFI
In CPUs section, set CPU model to host-passthrough, and CPU Topology to whatever fits your system.
For SATA disk of VM, set Disk Bus to virtio.
In NIC section, set Device Model to virtio
Add Hardware > CDROM: virtio-win.iso
Now, Begin Installation. Windows can't detect the virtio disk, so you need to Load Driver and select virtio-iso/amd64/win10 when prompted.
After successful installation of Windows, install virtio drivers from virtio CDROM. You can then remove virtio iso.

Attaching PCI devices

Remove Channel Spice, Display Spice, Video QXL, Sound ich* and other unnecessary devices.
Now, click on Add Hardware, select PCI Devices and add the PCI Host devices for your GPU's VGA and HDMI Audio.

Libvirt Hooks

Libvirt hooks automate the process of running specific tasks during VM state change.
More info at: PassthroughPost

Note: Comment Unbind/rebind EFI framebuffer line from start and stop script if you're using AMD 6000 series cards (#9). Also, move the line to unload AMD kernal module below detaching devices from host. These might also apply to older AMD cards.

Create Libvirt Hook

mkdir /etc/libvirt/hooks
touch /etc/libvirt/hooks/qemu
chmod +x /etc/libvirt/hooks/qemu

/etc/libvirt/hooks/qemu

#!/bin/bash

GUEST_NAME="$1"
HOOK_NAME="$2"
STATE_NAME="$3"
MISC="${@:4}"

BASEDIR="$(dirname $0)"

HOOKPATH="$BASEDIR/qemu.d/$GUEST_NAME/$HOOK_NAME/$STATE_NAME"
set -e # If a script exits with an error, we should as well.

if [ -f "$HOOKPATH" ]; then
eval \""$HOOKPATH"\" "$@"
elif [ -d "$HOOKPATH" ]; then
while read file; do
  eval \""$file"\" "$@"
done <<< "$(find -L "$HOOKPATH" -maxdepth 1 -type f -executable -print;)"
fi

Create Start Script

mkdir -p /etc/libvirt/hooks/qemu.d/win10/prepare/begin
touch /etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh
chmod +x /etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh

Note: If you're on KDE Plasma (Wayland), you need to terminate user services alongside display-manager (#31).

/etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh

#!/bin/bash
set -x

# Stop display manager
systemctl stop display-manager
# systemctl --user -M YOUR_USERNAME@ stop plasma*

# Unbind VTconsoles: might not be needed
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind

# Unbind EFI Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

# Unload NVIDIA kernel modules
modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia

# Unload AMD kernel module
# modprobe -r amdgpu

# Detach GPU devices from host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-detach pci_0000_01_00_0
virsh nodedev-detach pci_0000_01_00_1

# Load vfio module
modprobe vfio-pci

Create Stop Script

mkdir -p /etc/libvirt/hooks/qemu.d/win10/release/end
touch /etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh
chmod +x /etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh

/etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh

#!/bin/bash
set -x

# Attach GPU devices to host
# Use your GPU and HDMI Audio PCI host device
virsh nodedev-reattach pci_0000_01_00_0
virsh nodedev-reattach pci_0000_01_00_1

# Unload vfio module
modprobe -r vfio-pci

# Load AMD kernel module
#modprobe amdgpu

# Rebind framebuffer to host
echo "efi-framebuffer.0" > /sys/bus/platform/drivers/efi-framebuffer/bind

# Load NVIDIA kernel modules
modprobe nvidia_drm
modprobe nvidia_modeset
modprobe nvidia_uvm
modprobe nvidia

# Bind VTconsoles: might not be needed
echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > /sys/class/vtconsole/vtcon1/bind

# Restart Display Manager
systemctl start display-manager

Keyboard/Mouse Passthrough

In order to be able to use keyboard/mouse in the VM, you can either passthrough the USB Host device or use Evdev passthrough.

Using USB Host Device is simple,
Add Hardware > USB Host Device, add your keyboard and mouse device.

For Evdev passthrough, follow these steps:
Modify libvirt configuration of your VM.
Note: Save only after adding keyboard and mouse devices or the changes gets lost.
Change first line to:

virsh edit win10
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

Find your keyboard and mouse devices in /dev/input/by-id. You'd generally use the devices ending with event-kbd and event-mouse. And the devices in your configuration right before closing </domain> tag.
Replace MOUSE_NAME and KEYBOARD_NAME with your device id.

virsh edit win10

...
  <qemu:commandline>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=mouse1,evdev=/dev/input/by-id/MOUSE_NAME'/>
    <qemu:arg value='-object'/>
    <qemu:arg value='input-linux,id=kbd1,evdev=/dev/input/by-id/KEYBOARD_NAME,grab_all=on,repeat=on'/>
  </qemu:commandline>
</domain>

You need to include these devices in your qemu config.

/etc/libvirt/qemu.conf

...
user = "YOUR_USERNAME"
group = "kvm"
...
cgroup_device_acl = [
    "/dev/input/by-id/KEYBOARD_NAME",
    "/dev/input/by-id/MOUSE_NAME",
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/sev"
]
...

Also, switch from PS/2 devices to virtio devices. Add the devices inside <devices> block

virsh edit win10
... <devices> ... <input type='mouse' bus='virtio'/> <input type='keyboard' bus='virtio'/> ... </devices> ...

Audio Passthrough

VM's audio can be routed to the host using Pipewire or Pulseaudio.
You can also use Scream instead.

Pipewire

From ArchWiki

You need to have Pipewire with JACK support.

Note: You may use Carla to figure out appropriate input/output. Replace 1000 with your current user id.

virsh edit win10

...
  <devices>
    ...
    <audio id="1" type="jack">
      <input clientName="win10" connectPorts="your-input"/>
      <output clientName="win10" connectPorts="your-output"/>
    </audio>
  </devices>
  <qemu:commandline>
    <qemu:env name="PIPEWIRE_RUNTIME_DIR" value="/run/user/1000"/>
    <qemu:env name="PIPEWIRE_LATENCY" value="512/48000"/>
  </qemu:commandline>
</domain>

Pulseaudio

Note: Replace 1000 with your current user id.

virsh edit win10

...
  <qemu:commandline>
    ...
    <qemu:arg value="-device"/>
    <qemu:arg value="ich9-intel-hda,bus=pcie.0,addr=0x1b"/>
    <qemu:arg value="-device"/>
    <qemu:arg value="hda-micro,audiodev=hda"/>
    <qemu:arg value="-audiodev"/>
    <qemu:arg value="pa,id=hda,server=/run/user/1000/pulse/native"/>
  </qemu:commandline>
</domain>

Video card driver virtualisation detection

Video Card drivers refuse to run in Virtual Machine, so you need to spoof Hyper-V Vendor ID.

virsh edit win10
... <features> ... <hyperv> ... <vendor_id state='on' value='whatever'/> ... </hyperv> ... </features> ...

NVIDIA guest drivers also require hiding the KVM CPU leaf:

virsh edit win10
... <features> ... <kvm> <hidden state='on'/> </kvm> ... </features> ...

vBIOS Patching

NOTE: You are only making changes on dumped ROM file. Your hardware is safe.
While most of the GPU can be passed with stock vBIOS, some GPU requires vBIOS patching to passthrough.
In order to patch vBIOS, you need to first dump the GPU vBIOS from your system.
If you have Windows installed, you can use GPU-Z to dump vBIOS.
To dump vBIOS on Linux, you can use following command (replace PCI id with yours):

echo 1 > /sys/bus/pci/devices/0000:01:00.0/rom
cat /sys/bus/pci/devices/0000:01:00.0/rom > path/to/dump/vbios.rom
echo 0 > /sys/bus/pci/devices/0000:01:00.0/rom

If you're not in root shell, you should use the above commands with sudo as:

echo 1 | sudo tee /sys/bus/pci/devices/0000:01:00.0/rom
sudo cat /sys/bus/pci/devices/0000:01:00.0/rom > path/to/dump/vbios.rom
echo 0 | sudo tee /sys/bus/pci/devices/0000:01:00.0/rom

To patch vBIOS, you need to use Hex Editor (eg., Okteta) and trim unnecessary header.
For NVIDIA GPU, using hex editor, search string “VIDEO”, and remove everything before HEX value 55.
I'm not sure about AMD, but the process should be similar.

To use patched vBIOS, edit VM's configuration to include patched vBIOS inside hostdev block of VGA

virsh edit win10
... <hostdev mode='subsystem' type='pci' managed='yes'> <source> ... </source> <rom file='/home/me/patched.rom'/> ... </hostdev> ...

Name		Name	Last commit message	Last commit date
Latest commit History 73 Commits
README.md		README.md
libvirt_hooks_setup.sh		libvirt_hooks_setup.sh

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

Table Of Contents

Enable & Verify IOMMU

Install required tools

Enable required services

Setup Guest OS

Attaching PCI devices

Libvirt Hooks

Keyboard/Mouse Passthrough

Audio Passthrough

Video card driver virtualisation detection

vBIOS Patching

See Also

About

Contributors 5

Languages

QaidVoid/Complete-Single-GPU-Passthrough

Folders and files

Latest commit

History

Repository files navigation

Table Of Contents

Enable & Verify IOMMU

Install required tools

Enable required services

Setup Guest OS

Attaching PCI devices

Libvirt Hooks

Keyboard/Mouse Passthrough

Audio Passthrough

Video card driver virtualisation detection

vBIOS Patching

See Also

About

Topics

Resources

Stars

Watchers

Forks

Contributors 5

Languages