Compiling/Installing NVIDIA Drivers for Talos

[Arc676]

Are there any reference materials available for building NVIDIA drivers as a Talos system extension? I'm following the conclusion from this thread and building the NVIDIA system extension with vGPU support from scratch, but I'm running into some issues getting the installer to work with the tools described in the Talos kernel module documentation.

NVIDIA provides an installer program, nvidia-installer. To run on Alpine, it requires¹

• libfts, which is available via musl-fts-dev
• A couple string symbols from the standard library that are missing from musl, but these can easily be manually compiled and provided as a shared library
• The kernel source; there is linux-lts-dev, but I'm not sure if this corresponds to the correct source tree for the kernel used by Talos
• glibc, to some extent (not sure which parts, exactly)

The installer is working fine in a RHEL Docker image, but this isn't representative of the capabilities of the Alpine-based build environment. So far I've attempted to sidestep the limitations of the build environment by building a couple separate images and loading the installer's dependencies, but this seems to be a dead end (not to mention a terrible long-term solution anyway). The installer manages to launch, but fails to build or install anything:

ERROR: No package found for installation. Please run this utility with the '--help' option for usage information.

Given that glibc is available as a system extension, it would seem that musl compatibility isn't exactly a new topic. Cross-compilation had crossed my mind as an alternative, but I figured I'd ask here first to see if there was already a suggested solution.

What is the recommended approach for compiling drivers for Talos, such that these can be added to the image as an extension, when the driver installer depends on glibc?

Package Definition

Contents of nvidia-vgpu/pkg.yaml following structure described in the documentation:

name: nvidia-vgpu-pkg
variant: alpine      # base container for environment (e.g. alpine, scratch)
shell: /bin/sh        # shell to use to execute commands in steps
dependencies:         # other steps required before building package
  - stage: base
  - image: ghcr.io/siderolabs/glibc:2.41 # borrowing glibc from the system extension
    to: /glibc
  - image: private-registry.com/image-with-libraries
    to: /vgpu-libs
steps:                # steps needed to build package container
  - prepare:
      - ln -s /vgpu-libs/lib64/ld-linux-x86-64.so.2 /lib64/ld-linux-x86-64.so.2 # set up compatibility library
      - ldd /pkg/patches/nvidia/nvidia-installer >&2 # verify that external symbols are defined
      - PATH=$PATH:/glibc/rootfs/usr/sbin:/glibc/rootfs/usr/bin:/glibc/rootfs/bin:/vgpu-libs/sbin /pkg/patches/nvidia/nvidia-installer
          --ui=none
          --kernel-source-path=/vgpu-libs/src/linux-headers-6.12.45-0-lts
          --no-rebuild-initramfs
          --no-systemd
          --no-install-compat32-libs || cat /var/log/nvidia-installer.log >&2
    env:
      LD_PRELOAD: /vgpu-libs/libfts.so:/vgpu-libs/additional-symbols.so
      LD_LIBRARY_PATH: /glibc/lib:/glibc/lib64
  - test:             # validate software
      - nvidia-smi || true
finalize:             # copy directory structure from source to destination
  - from: /pkg/manifest.yaml
    to: /

Footnotes

1. Based on ldd output and the program's own messages ↩

[frezbo]

You can refer to https://github.com/siderolabs/pkgs/blob/main/nonfree/kmod-nvidia/production/pkg.yaml oh how we build the modules, building modules doesn't need glibc at all

And the other part of nvidia stuff here https://github.com/siderolabs/extensions/blob/main/nvidia-gpu/nvidia-container-toolkit/nvidia-container-cli/production/pkg.yaml which uses glibc

[Arc676]

Thanks; I had consulted these before, but I got tunnel vision on the installer script that had been provided and didn't think of compiling from source directly. I managed to compile the NVIDIA driver by using that first module as a reference.

Unfortunately NVIDIA's kernel module isn't compatible with the current Linux kernel due to its dependency on some outdated symbols, but that's a different topic.