The NVIDIA ICD JSON occasionally goes missing from 'nvidia-ctk cdi generate'

[debarshiray]

I have been playing with the NVIDIA Container Toolkit on Fedora 39 Workstation and the proprietary NVIDIA driver from RPM Fusion. I have noticed that the NVIDIA installable client driver (or ICD) JSON for Vulkan occasionally goes missing from nvidia-ctk cdi generate:

$ nvidia-ctk cdi generate --format yaml 2>/dev/null | grep vulkan
 - containerPath: /etc/vulkan/implicit_layer.d/nvidia_layers.json
   hostPath: /usr/share/vulkan/implicit_layer.d/nvidia_layers.json

... even though the file is present on the host operating system at /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json and Vulkan support on the host is confirmed by:

$ vulkaninfo --summary
...
...
Devices:
========
GPU0:
	apiVersion         = 1.3.280
	driverVersion      = 560.35.3.0
	vendorID           = 0x10de
	deviceID           = 0x1cbc
	deviceType         = PHYSICAL_DEVICE_TYPE_DISCRETE_GPU
	deviceName         = Quadro P600
	driverID           = DRIVER_ID_NVIDIA_PROPRIETARY
	driverName         = NVIDIA
	driverInfo         = 560.35.3.0
	conformanceVersion = 1.3.8.2
	deviceUUID         = 2efa4848-ba99-ccd3-0a19-f497b31331ca
	driverUUID         = c3ca0510-c7e6-5f1c-86a1-dc0ed4ea4e21
...
...

This means that Podman containers don't have Vulkan support through the proprietary NVIDIA driver, and can only use LLVMpipe.

Right now, I am observing this problem with:

$ uname --kernel-release
6.11.4-101.fc39.x86_64
$ rpm -q kernel
kernel-6.5.6-300.fc39.x86_64
kernel-6.11.4-101.fc39.x86_64
$ rpm -q kmod-nvidia
kmod-nvidia-560.35.03-1.fc39.x86_64

[debarshiray]

I forgot to mention the NVIDIA Container Toolkit version:

$ nvidia-ctk --version
NVIDIA Container Toolkit CLI version 1.16.1
$ rpm -qf $(which nvidia-ctk)
golang-github-nvidia-container-toolkit-1.16.1-1.fc39.x86_64

Note that the NVIDIA Container Toolkit version didn't change between the NVIDIA ICD JSON for Vulkan being listed and not listed. What changed was that I pulled in the RPM updates for the rest of the Fedora host.

[elezar]

@debarshiray the host path you mention /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json is not one that we explicitly search for. Could you please confirm which package provides that file? It could be that the 560.35.3.0 driver that you're using now includes the file including the architecture string.

(Looking at some older internal documentation it seems as if this has been the case for a while).

[debarshiray]

Thanks for looking into it, @elezar !

Meanwhile, I reinstalled different versions of Fedora a few times to see if the problem is specific to a particular combination of package versions. I could reproduce it reliably on Fedora 40 and 41, which was surprising because this used to work. :)

Now with Fedora 41 Workstation and the proprietary NVIDIA driver from RPM Fusion, I see:

$ rpm --query --file /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json
xorg-x11-drv-nvidia-libs-560.35.03-5.fc41.x86_64

If I force /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json to be present inside the container through an explicit bind mount then I do get Vulkan support through the proprietary NVIDIA driver.

In all cases, Vulkan support is available through the proprietary driver on the host operating system, as shown in the vulkaninfo --summary snippet above.

[elezar]

Who is the publisher of the xorg-x11-drv-nvidia-libs-560.35.03-5.fc41.x86_64 package above?

[debarshiray]

Who is the publisher of the xorg-x11-drv-nvidia-libs-560.35.03-5.fc41.x86_64 package above?

It's RPM Fusion. That's where I got the proprietary NVIDIA driver from.

[elezar]

The issue is that the driver package definition (see https://pkgs.rpmfusion.org/cgit/nonfree/xorg-x11-drv-nvidia.git/tree/xorg-x11-drv-nvidia.spec#n294) changes the name that the NVIDIA tooling expects. This means that the NVIDIA Container Toolkit can't locate the expected ICD.

My suggestion would be to create a bug against RPM Fusion so as to maintain the behaviour of the NVIDIA driver.

As a workaround you could rename / copy the nvidia_icd.x86_64.json file to nvidia_icd.json instead or ensure that you start your containers with:

-v /usr/share/vulkan/icd.d/nvidia_icd.x86_64.json:/usr/share/vulkan/icd.d/nvidia_icd.json

or adding an additional mount to your CDI spec.

I will look into what would be required for a more stable workaround, but can't commit to a specific timeline.

[debarshiray]

The issue is that the driver package definition (see https://pkgs.rpmfusion.org/cgit/nonfree/xorg-x11-drv-nvidia.git/tree/xorg-x11-drv-nvidia.spec#n294) changes the name that the NVIDIA tooling expects. This means that the NVIDIA Container Toolkit can't locate the expected ICD.

I see. Thanks for tracking that down.

My suggestion would be to create a bug against RPM Fusion so as to maintain the behaviour of the NVIDIA driver.

Done: https://bugzilla.rpmfusion.org/show_bug.cgi?id=7167

As a workaround you could rename / copy the nvidia_icd.x86_64.json file to nvidia_icd.json instead or ensure that you start your containers with:

I created a symbolic link and it does indeed work.

I will look into what would be required for a more stable workaround, but can't commit to a specific timeline.

Okay! My apologies for adding yet another thing to your pile. :/

[kwizart]

RPM Fusion co-maintainer here.

The RPM Fusion change is legit and correspond to a move in the ICD loader specification where the expected pattern for the file is now vendor_icd.target_cpu.json . This is correctly handled by vulkaninfo and mesa drivers are also using the new pattern.

This avoid a conflict when both i686 and x86_64 vulkan libraries are installed on the same system for any other usage.

There is a workaround, but the fix is likely to start looking at nvidia_icd.target_cpu before nvidia_icd.json as a compatibility.

[debarshiray]

Thanks for the input, @kwizart !

[elezar]

Thanks for the update @kwizart (and sorry for the delay). Let me update our code to handle the target_cpu case as well.