Changing configuration about containerd runtime on kubernetes looks weird

[SeungminHeo]

1. Quick Debug Information

• OS/Version(e.g. RHEL8.6, Ubuntu22.04): RHEL8.7
• Kernel Version: 4.18.0-477.15.1.el8_8.x86_64
• Container Runtime Type/Version(e.g. Containerd, CRI-O, Docker): Containerd
• K8s Flavor/Version(e.g. K8s, OCP, Rancher, GKE, EKS): Vanila K8s, v1.26.3
• GPU Operator Version: v23.3.2

2. Issue or feature description

I deployed GPU Operator on kubernetes cluster generated by kubespray with containerd.
I didn't pre-install any driver or container toolkit. I deployed them with GPU Operator

What i want to know is that whether updating on /etc/containerd/config.toml can affect running pods and provisioning pods or not.
After updating my kubernetes setting (especially docker registry options), all the contents in /etc/containerd/config.toml was overriden . All contents related with nvidia container runtime disappeared. Fortunately, running pods were not affected. And newly provisioned pods were also not affected.

If I restart nvidia toolkit daemonset, contents will be changed to nvidia container runtime related, but I can't figure out this is normal situation. Is it normal and okay? Can I leave config.toml as it is or should I update it with restarting container toolkit?

3. Steps to reproduce the issue

• Update kubernetes configuration, especially on conatinerd runtime. (/etc/containerd/config.toml)

4. Information to attach (optional if deemed irrelevant)

• two config.toml file changed (other configs like registry removed)

Before

oom_score = 0
root = "/DATA/containerd"
state = "/run/containerd"
version = 2
 
[debug]
  level = "info"
 
[grpc]
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
 
[metrics]
  address = ""
  grpc_histogram = false
 
[plugins]
 
  [plugins."io.containerd.grpc.v1.cri"]
    enable_unprivileged_icmp = false
    enable_unprivileged_ports = false
    max_container_log_line_size = -1
    sandbox_image = "registry.k8s.io/pause:3.8"
 
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "nvidia"
      snapshotter = "overlayfs"
 
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
 
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
 
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime"
            systemdCgroup = true
 
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-cdi]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
 
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-cdi.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.cdi"
            systemdCgroup = true
 
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-experimental]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
 
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-experimental.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.experimental"
            systemdCgroup = true
 
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-legacy]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
 
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia-legacy.options]
            BinaryName = "/usr/local/nvidia/toolkit/nvidia-container-runtime.legacy"
            systemdCgroup = true
 
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          base_runtime_spec = "/etc/containerd/cri-base.json"
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"
 
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            systemdCgroup = true

After (overriden)

version = 2
root = "/DATA/containerd"
state = "/run/containerd"
oom_score = 0
 
[grpc]
  max_recv_message_size = 16777216
  max_send_message_size = 16777216
 
[debug]
  level = "info"
 
[metrics]
  address = ""
  grpc_histogram = false
 
[plugins]
  [plugins."io.containerd.grpc.v1.cri"]
    sandbox_image = "registry.k8s.io/pause:3.8"
    max_container_log_line_size = -1
    enable_unprivileged_ports = false
    enable_unprivileged_icmp = false
    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      snapshotter = "overlayfs"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
          runtime_type = "io.containerd.runc.v2"
          runtime_engine = ""
          runtime_root = ""
          base_runtime_spec = "/etc/containerd/cri-base.json"
 
          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            systemdCgroup = true

[SeungminHeo]

Further findings are below.

• Containerd process is restarted after apply updating kubernetes config and overriding /etc/containerd/config.toml
• After /etc/containerd/config.toml overriden, nvidia-smi command inside newly provisioned pod works fine. Also, torch.cuda.is_available() returns True.
• Newly provisioned pod uses cpu. Already provisioned pod don't use cpu at all.
• Also, I ran gpu-operator-validator and failed.

Then, should i always restart container toolkit after update kubernetes update?

[shivamerla]

@SeungminHeo yes, if /etc/containerd/config.toml is overwritten by any means, then container-toolkit container have to be restarted to re-apply nvidia runtime configuration. Without this new GPU pods will fail. container-toolkit always applies the config and restarts containerd. Already running pods will not be affected, but new ones are.

[SeungminHeo]

Thank you for your explanation 👍 Our team has to follow your instruction.