feat: nvidia-fabricmanager support Blackwell baseboards (DGX/HGX B100/B200/B300)

Quoting: https://docs.nvidia.com/datacenter/tesla/pdf/fabric-manager-user-guide.pdf

> On NVIDIA DGX-B200, HGX-B200, and HGX-B100 systems and later, the FabricManager package needs an additional NVLSM dependency for proper operation.
> NVLink Subnet manager (NVLSM) originated from the InfiniBand networking and contains additional logic to program NVSwitches and NVLinks.

Not running the NVLSM on Blackwell and newer fabrics with NVLink 5.0+ will result in FabricManager failing to start with error `NV_WARN_NOTHING_TO_DO`. NVSwitches will remain uninitialized and applications will fail with the `CUDA_ERROR_SYSTEM_NOT_READY` or `cudaErrorSystemNotReady` error.  The CUDA initialization process can only begin after the GPUs complete their registration process with the NVLink fabric.
A GPU fabric registration status can be verified with the command: `nvidia-smi -q -i 0 | grep -i -A 2 Fabric`. An `In Progress` state indicates that the GPU is being registered and FabricManager is likely not running or missing the NVLSM dependency. A `Completed` state is shown when the GPU is successfully registered with the NVLink fabric.

The FabricManager package includes the script `nvidia-fabricmanager-start.sh`, which is used to selectively start FabricManager and NVLSM processes depending on the underlying platform.
A key aspect of determining whether an NVLink 5.0+ fabric is present is to look for a Limited Physical Function (LPF) port in InfiniBand devices. To differentiate LPFs, the Vital Product Data (VPD) information includes a vendor-specific field called `SMDL`, with a non-zero value defined as `SW_MNG`. The first device is then selected, and it's port GUID is extracted and passed to NVLSM and FabricManager.
So both services share a configuration key that result from a common initialization process. Additionally, they communicate with each other over a Unix socket.

This patch introduces the following changes:
 * Adds NVLSM to the nvidia-fabricmanager extension.
 * Introduces a new `nvidia-fabricmanager-wrapper` program to replicate the initialization process from `nvidia-fabricmanager-start.sh`:
   * Detects NVLink 5.0+ fabrics and extracts an NVSwitch LPF port GUID. This is done by calling libibumad directly with CGO instead of parsing the output of the `ibstat` command as in the upstream script.
   * Starts FabricManager, and NVLSM only when needed.
   * Keeps both process lifecycles synchronized and ensures the Talos container will restart if either process crashes.
 * Refactors the nvidia-fabricmanager container to be self-contained, as this service does not share files with other nvidia-gpu extensions.

Signed-off-by: Thibault VINCENT <[email protected]>
Signed-off-by: Noel Georgi <[email protected]>

Author: npdgm

deps.svg

@@ -5,4 +5,5 @@ use (
 	./examples/hello-world-service/src
 	./nvidia-gpu/nvidia-container-toolkit/nvidia-container-runtime-wrapper
 	./nvidia-gpu/nvidia-container-toolkit/nvidia-persistenced-wrapper
+	./nvidia-gpu/nvidia-fabricmanager/nvidia-fabricmanager-wrapper
 )

go.work

@@ -1,10 +1,7 @@
 # https://docs.nvidia.com/datacenter/tesla/pdf/fabric-manager-user-guide.pdf
 name: nvidia-fabricmanager
 container:
-  entrypoint: /usr/local/bin/nv-fabricmanager
-  args:
-   - --config
-   - /usr/local/share/nvidia/nvswitch/fabricmanager.cfg
+  entrypoint: /usr/bin/nvidia-fabricmanager-wrapper
   mounts:
     # device files
     - source: /dev
@@ -28,44 +25,40 @@ container:
       options:
         - bind
         - ro
-    # nvidia libraries
-    - source: /usr/local/lib
-      destination: /usr/local/lib
-      type: bind
-      options:
-        - bind
-        - ro
     # service state file
+    # * nvlsm:
+    #   - pid file that can't be disabled
+    #   - unix socket /var/run/nvidia-fabricmanager/fm_sm_ipc.socket
+    #     can't be changed, path is hardcoded into fabricmanager
+    # * fabricmanager
+    #   - state file
+    #   - database files
     - source: /var/run/nvidia-fabricmanager
-      destination: /var/run/nvidia-fabricmanager
+      destination: /var/run
       type: bind
       options:
         - rshared
         - rbind
         - rw
-    # log files
-    - source: /var/log
-      destination: /var/log
+    # service cache file
+    # * nvlsm: database files
+    - source: /var/cache/nvidia-fabricmanager
+      destination: /var/cache
       type: bind
       options:
         - rshared
         - rbind
         - rw
-    # fabric topology files
-    - source: /usr/local/share/nvidia/nvswitch
-      destination: /usr/local/share/nvidia/nvswitch
+    # service log files
+    # * nvlsm:
+    #   - mandatory dump files hardcoded to /var/log/<file>, so /var/log must be writable
+    - source: /var/log/nvidia-fabricmanager
+      destination: /var/log
       type: bind
       options:
         - rshared
         - rbind
-        - ro
-    # binaries
-    - source: /usr/local/bin
-      destination: /usr/local/bin
-      type: bind
-      options:
-        - bind
-        - ro
+        - rw
 depends:
   - service: cri
   # we need to depend on udevd so that the nvidia device files are created

nvidia-gpu/nvidia-fabricmanager/lts/nvidia-fabricmanager.yaml

@@ -2,48 +2,79 @@ name: nvidia-fabricmanager-lts
 variant: scratch
 shell: /bin/bash
 dependencies:
- - stage: base
+  - stage: base
+  - stage: nvidia-fabricmanager-gcc-runtime
+  - stage: nvidia-fabricmanager-wrapper
 steps:
   - sources:
     # {{ if eq .ARCH "aarch64" }} This in fact is YAML comment, but Go templating instruction is evaluated by bldr
       - url: https://developer.download.nvidia.com/compute/nvidia-driver/redist/fabricmanager/linux-sbsa/fabricmanager-linux-sbsa-{{ .NVIDIA_DRIVER_LTS_VERSION }}-archive.tar.xz
         destination: fabricmanager.tar.xz
         sha256: {{ .NVIDIA_FABRIC_MANAGER_LTS_ARM64_SHA256 }}
         sha512: {{ .NVIDIA_FABRIC_MANAGER_LTS_ARM64_SHA512 }}
+      - url: https://developer.download.nvidia.com/compute/cuda/redist/nvlsm/linux-sbsa/nvlsm-linux-sbsa-{{ .NVIDIA_NVLSM_LTS_VERSION }}-archive.tar.xz
+        destination: nvlsm.tar.xz
+        sha256: {{ .NVIDIA_NVLSM_LTS_ARM64_SHA256 }}
+        sha512: {{ .NVIDIA_NVLSM_LTS_ARM64_SHA512 }}
     # {{ else }} This in fact is YAML comment, but Go templating instruction is evaluated by bldr
       - url: https://developer.download.nvidia.com/compute/nvidia-driver/redist/fabricmanager/linux-x86_64/fabricmanager-linux-x86_64-{{ .NVIDIA_DRIVER_LTS_VERSION }}-archive.tar.xz
         destination: fabricmanager.tar.xz
         sha256: {{ .NVIDIA_FABRIC_MANAGER_LTS_AMD64_SHA256 }}
         sha512: {{ .NVIDIA_FABRIC_MANAGER_LTS_AMD64_SHA512 }}
+      - url: https://developer.download.nvidia.com/compute/cuda/redist/nvlsm/linux-x86_64/nvlsm-linux-x86_64-{{ .NVIDIA_NVLSM_LTS_VERSION }}-archive.tar.xz
+        destination: nvlsm.tar.xz
+        sha256: {{ .NVIDIA_NVLSM_LTS_AMD64_SHA256 }}
+        sha512: {{ .NVIDIA_NVLSM_LTS_AMD64_SHA512 }}
     # {{ end }} This in fact is YAML comment, but Go templating instruction is evaluated by bldr
     prepare:
       - |
-        tar -xf fabricmanager.tar.xz --strip-components=1
+        mkdir fm sm
+        tar -xf fabricmanager.tar.xz --strip-components=1 -C fm
+        tar -xf nvlsm.tar.xz --strip-components=1 -C sm
     install:
       - |
-        mkdir -p /rootfs/usr/local/bin \
-          /rootfs/usr/local/lib \
-          /rootfs/usr/local/share/nvidia/nvswitch \
-          /rootfs/usr/local/lib/containers/nvidia-fabricmanager \
-          /rootfs/usr/local/etc/containers
+        mkdir -p /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/bin \
+                 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/lib \
+                 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch \
+                 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/opt/nvidia/nvlsm/sbin \
+                 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/opt/nvidia/nvlsm/lib \
+                 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvlsm
+      # nvlsm
+      - |
+        cp sm/sbin/nvlsm /rootfs/usr/local/lib/containers/nvidia-fabricmanager/opt/nvidia/nvlsm/sbin/
 
-        cp lib/libnvfm.so.1 /rootfs/usr/local/lib/libnvfm.so.1
-        ln -s libnvfm.so.1 /rootfs/usr/local/lib/libnvfm.so
+        cp sm/lib/libgrpc_mgr.so /rootfs/usr/local/lib/containers/nvidia-fabricmanager/opt/nvidia/nvlsm/lib/
 
-        cp bin/nv-fabricmanager /rootfs/usr/local/bin/
-        cp bin/nvswitch-audit /rootfs/usr/local/bin/
+        cp sm/share/nvidia/nvlsm/device_configuration.conf \
+           sm/share/nvidia/nvlsm/grpc_mgr.conf \
+           sm/share/nvidia/nvlsm/nvlsm.conf \
+           /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvlsm/
+      # fabricmanager
+      - |
+        cp fm/bin/nv-fabricmanager \
+           fm/bin/nvswitch-audit \
+           /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/bin/
 
-        cp share/nvidia/nvswitch/dgx2_hgx2_topology /rootfs/usr/local/share/nvidia/nvswitch/
-        cp share/nvidia/nvswitch/dgxa100_hgxa100_topology /rootfs/usr/local/share/nvidia/nvswitch/
+        cp fm/lib/libnvfm.so.1 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/lib/
+        ln -s libnvfm.so.1 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/lib/libnvfm.so
 
-        cp etc/fabricmanager.cfg  /rootfs/usr/local/share/nvidia/nvswitch/
+        cp fm/share/nvidia/nvswitch/* \
+           fm/etc/fabricmanager.cfg  \
+           /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/
 
-        cp /pkg/nvidia-fabricmanager.yaml /rootfs/usr/local/etc/containers/nvidia-fabricmanager.yaml
+        sed -i 's/DAEMONIZE=.*/DAEMONIZE=0/g' /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/fabricmanager.cfg
+        sed -i 's/LOG_FILE_NAME=.*/LOG_FILE_NAME=/g' /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/fabricmanager.cfg
+        sed -i 's#STATE_FILE_NAME=.*#STATE_FILE_NAME=/var/run/fabricmanager.state#g' /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/fabricmanager.cfg
 
-        sed -i 's/DAEMONIZE=.*/DAEMONIZE=0/g' /rootfs/usr/local/share/nvidia/nvswitch/fabricmanager.cfg
-        sed -i 's/STATE_FILE_NAME=.*/STATE_FILE_NAME=\/var\/run\/nvidia-fabricmanager\/fabricmanager.state/g' /rootfs/usr/local/share/nvidia/nvswitch/fabricmanager.cfg
-        sed -i 's/TOPOLOGY_FILE_PATH=.*/TOPOLOGY_FILE_PATH=\/usr\/local\/share\/nvidia\/nvswitch/g' /rootfs/usr/local/share/nvidia/nvswitch/fabricmanager.cfg
-        sed -i 's/DATABASE_PATH=.*/DATABASE_PATH=\/usr\/local\/share\/nvidia\/nvswitch/g' /rootfs/usr/local/share/nvidia/nvswitch/fabricmanager.cfg
+        if grep -q '^DATABASE_PATH=' /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/fabricmanager.cfg
+        then
+          sed -i 's#DATABASE_PATH=.*#DATABASE_PATH=/var/run#g' /rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/fabricmanager.cfg
+        else
+          echo -e '\nDATABASE_PATH=/var/run\n' >>/rootfs/usr/local/lib/containers/nvidia-fabricmanager/usr/share/nvidia/nvswitch/fabricmanager.cfg
+        fi
+      - |
+        mkdir -p /rootfs/usr/local/etc/containers
+        cp /pkg/nvidia-fabricmanager.yaml /rootfs/usr/local/etc/containers/nvidia-fabricmanager.yaml
     test:
       - |
         mkdir -p /extensions-validator-rootfs

nvidia-gpu/nvidia-fabricmanager/lts/pkg.yaml

@@ -0,0 +1,12 @@
+name: nvidia-fabricmanager-gcc-runtime
+variant: scratch
+dependencies:
+  - image: cgr.dev/chainguard/wolfi-base@{{ .WOLFI_BASE_REF }}
+steps:
+  - install:
+      - |
+        mkdir -p /rootfs/usr/local/lib/containers/nvidia-fabricmanager/opt/nvidia/nvlsm/lib
+        cp /usr/lib/libgcc_s.so.1 /rootfs/usr/local/lib/containers/nvidia-fabricmanager/opt/nvidia/nvlsm/lib/
+finalize:
+  - from: /rootfs
+    to: /rootfs

nvidia-gpu/nvidia-fabricmanager/nvidia-fabricmanager-gcc-runtime/pkg.yaml

@@ -0,0 +1 @@
+INTERNAL_PACKAGE: true

nvidia-gpu/nvidia-fabricmanager/nvidia-fabricmanager-gcc-runtime/vars.yaml

@@ -0,0 +1,5 @@
+module nvidia-fabricmanager-wrapper
+
+go 1.23.0
+
+require github.com/goaux/decowriter v1.0.0

nvidia-gpu/nvidia-fabricmanager/nvidia-fabricmanager-wrapper/go.mod

@@ -0,0 +1,2 @@
+github.com/goaux/decowriter v1.0.0 h1:f1mfBWGFIo3Upev3gswfGLQzQvC4SBVYi2ZAkNZsIaU=
+github.com/goaux/decowriter v1.0.0/go.mod h1:8GKUmiBlNCYxVHU2vlZoQHwLvYh7Iw1c7/tRekJbX7o=

nvidia-gpu/nvidia-fabricmanager/nvidia-fabricmanager-wrapper/go.sum

@@ -0,0 +1,156 @@
+// This Source Code Form is subject to the terms of the Mozilla Public
+// License, v. 2.0. If a copy of the MPL was not distributed with this
+// file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+package main
+
+import (
+	"bufio"
+	"context"
+	"fmt"
+	"log"
+	"os"
+	"os/exec"
+	"os/signal"
+	"path/filepath"
+	"strings"
+	"sync"
+	"syscall"
+	"time"
+
+	"github.com/goaux/decowriter"
+)
+
+const (
+	// FabricManager
+	fmCmdFile     = "/usr/bin/nv-fabricmanager"
+	fmConfigFile  = "/usr/share/nvidia/nvswitch/fabricmanager.cfg"
+	fmStopTimeout = 5 * time.Second
+
+	// NVLSM
+	smCmdFile     = "/opt/nvidia/nvlsm/sbin/nvlsm"
+	smConfigFile  = "/usr/share/nvidia/nvlsm/nvlsm.conf"
+	smPidFile     = "/var/run/nvlsm.pid"
+	smSocket      = "/var/run/nvidia-fabricmanager/fm_sm_ipc.socket"
+	smStopTimeout = 5 * time.Second
+	smSocketWait  = 15 * time.Second
+)
+
+func runCommand(ctx context.Context, wg *sync.WaitGroup, doneCb func(), waitDelay time.Duration, path string, arg ...string) {
+	wg.Add(1)
+
+	cmd := exec.CommandContext(ctx, path, arg...)
+	cmd.WaitDelay = waitDelay
+	cmd.Cancel = func() error {
+		return cmd.Process.Signal(os.Interrupt)
+	}
+
+	// TODO line writer to log module
+	name := filepath.Base(path)
+	cmd.Stdout = decowriter.New(bufio.NewWriter(os.Stdout), []byte(name+": "), []byte{})
+	cmd.Stderr = decowriter.New(bufio.NewWriter(os.Stderr), []byte(name+": "), []byte{})
+
+	go func() {
+		log.Printf("nvidia-fabricmanager-wrapper: running command: %s %s\n", path, strings.Join(arg, " "))
+
+		err := cmd.Run()
+		if err == nil {
+			log.Printf("nvidia-fabricmanager-wrapper: command %s [%d] completed successfully\n", path, cmd.Process.Pid)
+		} else if exitErr, ok := err.(*exec.ExitError); ok {
+			if exitErr.Exited() {
+				log.Printf("nvidia-fabricmanager-wrapper: command %s [%d] exited with code %d\n", path, exitErr.Pid(),
+					exitErr.ExitCode())
+			} else {
+				log.Printf("nvidia-fabricmanager-wrapper: command %s [%d] was terminated\n", path, exitErr.Pid())
+			}
+		} else {
+			log.Printf("nvidia-fabricmanager-wrapper: failed to run command %s: %v\n", path, err)
+		}
+
+		wg.Done()
+		doneCb()
+	}()
+}
+
+func waitForFile(ctx context.Context, filepath string, timeout time.Duration) error {
+	timer := time.NewTimer(timeout)
+	defer timer.Stop()
+
+	for {
+		select {
+		case <-ctx.Done():
+			return fmt.Errorf("parent context canceled: %w", ctx.Err())
+		case <-timer.C:
+			return fmt.Errorf("timeout waiting for file")
+		default:
+			if _, err := os.Stat(filepath); err == nil {
+				return nil
+			}
+			time.Sleep(100 * time.Millisecond)
+		}
+	}
+}
+
+func main() {
+	var cmdWg sync.WaitGroup
+
+	signal.Ignore(syscall.SIGHUP)
+
+	runCtx, gracefulShutdown := context.WithCancel(context.Background())
+
+	signalsChan := make(chan os.Signal, 1)
+	signal.Notify(signalsChan, os.Interrupt)
+	signal.Notify(signalsChan, syscall.SIGTERM)
+
+	go func() {
+		received := <-signalsChan
+		signal.Stop(signalsChan)
+		log.Printf("nvidia-fabricmanager-wrapper: received signal '%s', initiating a graceful shutdown\n", received.String())
+		gracefulShutdown()
+	}()
+
+	nvswitchPorts := findNvswitchMgmtPorts()
+	for _, port := range nvswitchPorts {
+		log.Printf("nvidia-fabricmanager-wrapper: found NVSwitch LPF: device=%s guid=0x%x\n", port.IBDevice, port.PortGUID)
+	}
+
+	fmSmMgmtPortGUID := ""
+	if len(nvswitchPorts) > 0 {
+		fmSmMgmtPortGUID = fmt.Sprintf("0x%x", nvswitchPorts[0].PortGUID)
+		log.Printf("nvidia-fabricmanager-wrapper: using NVSwitch management port GUID: %s\n", fmSmMgmtPortGUID)
+	} else {
+		log.Println("nvidia-fabricmanager-wrapper: No InfiniBand NVSwitch detected. On Blackwell HGX baseboards and newer",
+			"with NVLink 5.0+, please load kernel module 'ib_umad' for NVLSM to run along FabricManager. Otherwise it will",
+			"fail to start with error NV_WARN_NOTHING_TO_DO, and GPU workloads will report CUDA_ERROR_SYSTEM_NOT_READY.")
+	}
+
+	if fmSmMgmtPortGUID != "" {
+		if err := os.Mkdir(filepath.Dir(smSocket), 0755); err != nil {
+			log.Printf("nvidia-fabricmanager-wrapper: error creating socket directory: %v\n", err)
+		}
+
+		runCommand(runCtx, &cmdWg, gracefulShutdown, smStopTimeout, smCmdFile, "--config", smConfigFile,
+			"--guid", fmSmMgmtPortGUID, "--pid_file", smPidFile, "--log_file", "stdout")
+
+		// vendor startup script waits for 5 seconds for NVLSM socket to be available before starting FM
+		// let's wait for the actual GRPC socket to be created by the plugin
+		log.Println("nvidia-fabricmanager-wrapper: waiting for socket creation at", smSocket)
+		err := waitForFile(runCtx, smSocket, smSocketWait)
+		if err != nil {
+			log.Printf("nvidia-fabricmanager-wrapper: error waiting for socket: %v\n", err)
+		} else {
+			log.Println("nvidia-fabricmanager-wrapper: socket found at", smSocket)
+		}
+		// for safety
+		time.Sleep(time.Second)
+	}
+
+	fmCmdArgs := []string{"--config", fmConfigFile}
+	if fmSmMgmtPortGUID != "" {
+		fmCmdArgs = append(fmCmdArgs, "--fm-sm-mgmt-port-guid", fmSmMgmtPortGUID)
+	}
+	runCommand(runCtx, &cmdWg, gracefulShutdown, fmStopTimeout, fmCmdFile, fmCmdArgs...)
+
+	log.Println("nvidia-fabricmanager-wrapper: initialization completed")
+	cmdWg.Wait()
+}