"DMA mapping failed" when trying passthrough with looking-glass

[srett]

Hey everyone, I'm not sure if this is even related to edk2/OVMF at all, but hopefully I might at least get some pointers here how/where to debug further.

I'm on an Alderlake platform, trying to pass through an nvidia T400 to a qemu VM. However, this is the output I get:

2023-07-07T12:04:52.153602Z qemu: VFIO_MAP_DMA failed: Das Argument ist ungültig
2023-07-07T12:04:52.153632Z qemu: vfio_dma_map(0x55c2dc872b60, 0x385000000000, 0x10000000, 0x7f8e88000000) = -22 (Das Argument ist ungültig)
2023-07-07T12:04:52.153902Z qemu: VFIO_MAP_DMA failed: Das Argument ist ungültig
2023-07-07T12:04:52.153910Z qemu: vfio_dma_map(0x55c2dc872b60, 0x385010000000, 0x2000000, 0x7f8e86000000) = -22 (Das Argument ist ungültig)
2023-07-07T12:04:52.331989Z qemu: VFIO_MAP_DMA failed: Das Argument ist ungültig
2023-07-07T12:04:52.332009Z qemu: vfio_dma_map(0x55c2dc872b60, 0x385000000000, 0x10000000, 0x7f8e88000000) = -22 (Das Argument ist ungültig)
2023-07-07T12:04:52.335508Z qemu: VFIO_MAP_DMA failed: Das Argument ist ungültig
2023-07-07T12:04:52.335535Z qemu: vfio_dma_map(0x55c2dc872b60, 0x385010000000, 0x2000000, 0x7f8e86000000) = -22 (Das Argument ist ungültig)
2023-07-07T12:04:52.405937Z qemu: VFIO_MAP_DMA failed: Das Argument ist ungültig
2023-07-07T12:04:52.405956Z qemu: vfio_dma_map(0x55c2dc872b60, 0x380800000000, 0x4000000, 0x7f8eac9ff000) = -22 (Das Argument ist ungültig)
qemu: hardware error: vfio: DMA mapping failed, unable to continue
CPU #0:
RAX=00000000b0000000 RBX=00000000b0808004 RCX=0000000000808004 RDX=0000000000000003
RSI=0000000000000003 RDI=000000007fef2680 RBP=000000007fef2490 RSP=000000007fef2430
R8 =0000000000000000 R9 =000000007fef258f R10=0000000000000000 R11=0000000000000004
R12=000000007fef267e R13=0000000000808000 R14=0000000000000003 R15=000000007fef267e
RIP=000000007ee8fb61 RFL=00000246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
CS =0038 0000000000000000 ffffffff 00a09b00 DPL=0 CS64 [-RA]
SS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
DS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
FS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
GS =0030 0000000000000000 ffffffff 00c09300 DPL=0 DS   [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0000 0000000000000000 0000ffff 00008b00 DPL=0 TSS64-busy
GDT=     000000007f9dc000 00000047
IDT=     000000007f3f9018 00000fff
CR0=80010033 CR2=0000000000000000 CR3=000000007fc01000 CR4=00000668
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000d00
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=8000000000000000 4007 FPR7=8000000000000000 4007
XMM00=0000000000000000 0000000000000000 XMM01=0000000000000000 0000000000000000
XMM02=0000000000000000 0000000000000000 XMM03=0000000000000000 0000000000000000
XMM04=0000000000000000 0000000000000000 XMM05=0000000000000000 0000000000000000
XMM06=0000000000000000 0000000000000000 XMM07=0000000000000000 0000000000000000
XMM08=0000000000000000 0000000000000000 XMM09=0000000000000000 0000000000000000
XMM10=0000000000000000 0000000000000000 XMM11=0000000000000000 0000000000000000
XMM12=0000000000000000 0000000000000000 XMM13=0000000000000000 0000000000000000
XMM14=0000000000000000 0000000000000000 XMM15=0000000000000000 0000000000000000
Abgebrochen

After googling around and ruling out issues other people had, mostly the memlock limit, I finally figured out the VM starts up successfully if I don't try to pass the shared memory block to the VM that is needed for getting the frame buffer out of the VM via looking glass. If I remove the shared memory, the VM boots, the GPU is available to the guest and can run 3D applications and CUDA, as can be verified by connecting a screen to the GPU directly.
Oddly enough, in that case I get even more lines like these on startup:

2023-07-07T12:04:52.153602Z qemu: VFIO_MAP_DMA failed: Das Argument ist ungültig
2023-07-07T12:04:52.153632Z qemu: vfio_dma_map(0x55c2dc872b60, 0x385000000000, 0x10000000, 0x7f8e88000000) = -22 (Das Argument ist ungültig)

Yet the VM and GPU appear to work flawlessly.

Another way to "fix" this, without removing the looking-glass shm, is to set the CPU to some specific (older) one, for example here Skylake.

This is my qemu command line, minus the HDD image:

/usr/bin/qemu-system-x86_64
-blockdev
{"driver":"file","filename":"/usr/share/qemu/edk2-x86_64-code.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}
-blockdev
{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}
-blockdev
{"driver":"file","filename":"/foo/bar/baz","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}
-blockdev
{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}
-machine
pc-q35-7.1,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format,hpet=off
-accel kvm
-m 1G
-cpu Skylake-Client
-object
{"qom-type":"memory-backend-ram","id":"pc.ram","size":1073741824}
-nodefaults
-device
{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}
-device
{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}
-device
{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}
-device
{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}
-device
{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}
-device
{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.7","addr":"0x0"}
-device
{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}
-device
{"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pcie.0","multifunction":true,"addr":"0x4"}
-device
{"driver":"vfio-pci","host":"0000:01:00.1","id":"hostdev1","bus":"pcie.0","addr":"0x4.0x1"}
-object
{"qom-type":"memory-backend-file","id":"shmmem-shmem0","mem-path":"/dev/shm/looking-glass","size":67108864,"share":true}
-device
{"driver":"ivshmem-plain","id":"shmem0","memdev":"shmmem-shmem0","bus":"pci.16","addr":"0x1"}
-display
gtk

I tried to debug this using qemu's gdb stub, and I can see that it seems to be executing at least some code from edk2/ovmf before the crash happens. Unfortunately, that doesn't drop me into gdb, qemu still crashes hard, I assume since gdb would only trap on errors in the guest, not if DMA mapping fails.

Also, removing edk2 from the qemu command line doesn't crash qemu, but instead it will just sit at the "no boot medium found" screen.

So I assume it has something to do with what edk2 does if the shm block is present, but also somehow with CPU feature flags!? I'm completely at a loss here, so I'm hoping someone familiar with the code base can at least point me in the right direction at where to look or what even to debug. Could this be something in edk2, or is it more likely a bug in qemu that edk2 just happens to trigger?

[mkfyi]

I ran into the same issue a few weeks ago, when I tried to set up a KVM with LookingGlass on Raptor Lake (13900K) for a good friend of mine and I couldn't fix it. The problem only seems to occur with newer CPUs that use efficiency and power cores because it works on Comet Lake (*), which was the last generation with only real (power) cores.

According to this communication via mail, someone is already working on a patch. Oh, I just found another very interesting mail from Eric Auger.

However, I'm not 100% sure if one or both patches will also solve the problem here.

(*) I can also verify that it works on Coffee Lake (Refresh).

Also worth mentioning: I also compiled and tried several stable releases by hand, because I read in some r/vfio thread that it should work with an older version of ovmf-edk2. Unfortunately, I couldn't confirm this.

[srett]

Yeah, we also tried BIOS updates and playing with different virtualization-related BIOS settings, to no avail.

Maybe the performance drop is due to Skylake being pretty old, but 50% sounds rough. We created a very hacky wrapper script for qemu-system-x86_64 that modifies the -cpu option generated by libvirt to set it to Cascade Lake and then additionally tries to enable every CPU feature flag qemu supports, since qemu luckily just complains if one is not supported by the host CPU, and keeps going without it. It also makes sure to set CPU model, family, stepping etc. to that of the host, in case software looks at those to pick specific optimizations.

It's to be placed next to the actual qemu binary (i.e. /usr/bin/qemu-system-x86_64.openslx), and then in the libvirt XML you need to point the /domain/devices/emulator tag to that script. I tested this on an older 9th gen machine, and at least in that case there appears to be no performance difference to just using -cpu host.

[srett]

So, @rafaelgieschke found a way easier fix :-)
-cpu host,host-phys-bits=false seems to work here. @mkfyi can you confirm this on your hw?

[rafaelgieschke]

For reference, the only directly visible effect of host-phys-bits=false is that it changes CPUID.80000008H:EAX from 0x0000302e (-cpu host) to 0x00003028 (-cpu host,host-phys-bits=false) in the guest.

[rafaelgieschke]

It also works with simply -cpu host,host-phys-bits-limit=0x28, which in libvirt >= 9.3.0 can be set using:

<cpu>
  <maxphysaddr mode='passthrough' limit='40'/>
</cpu>

[intrud0r]

It also works with simply -cpu host,host-phys-bits-limit=0x28, which in libvirt >= 9.3.0 can be set using:

<cpu>
  <maxphysaddr mode='passthrough' limit='40'/>
</cpu>

mkfyi is busy at the moment, so I will answer for him. Changing the CPU address size also fixed the hardware error from qemu for me without disabling any HyperV/CPU features in libvirt.
However, I had to set the limit in libvirt to 39 for the 13900K, any higher value results in a black screen with no error output.
Thanks a lot!

[dipankardas011]

but when I try to edit the configuration from the virtmanager xml section
it rewrites to

<cpu mode="custom" match="exact" check="none">
    <model fallback="forbid">qemu64</model>
    <maxphysaddr mode="passthrough"/>
  </cpu>

[nutta-git]

but when I try to edit the configuration from the virtmanager xml section it rewrites to

<cpu mode="custom" match="exact" check="none">
    <model fallback="forbid">qemu64</model>
    <maxphysaddr mode="passthrough"/>
  </cpu>

I have 12700k with ecores disabled and this worked for me (adjust topology to your liking),

<cpu mode="host-passthrough" check="none" migratable="on">
    <topology sockets="1" dies="1" clusters="1" cores="6" threads="2"/>
    <maxphysaddr mode="passthrough" limit="39"/>
  </cpu>

changing 40 to 39 made my vm boot