just update doing -f looks like fix the issue in the copy command

Coincidentally, we also saw this error crop up yesterday with one of our edge clusters after rebooting.

As an FYI i see different deployment yamls use different way to copy the cni binary in init container:

the first one[1] will use install_multus which will copy files in an atomic manner. the latter[2] will just use cp.
(install_multus support both thick and thin plugin types)

although im not sure that copying file atomically will solve the above issue.

see:
[1]

and
[2]
https://github.com/k8snetworkplumbingwg/multus-cni/blob/8e5060b9a7612044b7bf927365bbdbb8f6cde451/deployments/multus-daemonset-thick.yml#L199C9-L204C46

also deployments/multus-daemonset-crio.yml does not use init contianer.

This should hopefully be addressed with #1213

Saw this in minikube today. No rebooting, just staring up a new minikube cluster.

I also got a reproduction after rebooting a node and having multus restart.

I mitigated it by deleting /opt/cni/bin/multus-shim, but, yeah, I'll retest with the above patch

[fedora@labkubedualhost-master-1 whereabouts]$ watch -n1 kubectl get pods -A -o wide
[fedora@labkubedualhost-master-1 whereabouts]$ kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset-thick.yml
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io created
clusterrole.rbac.authorization.k8s.io/multus created
clusterrolebinding.rbac.authorization.k8s.io/multus created
serviceaccount/multus created
configmap/multus-daemon-config created
daemonset.apps/kube-multus-ds created
[fedora@labkubedualhost-master-1 whereabouts]$ watch -n1 kubectl get pods -A -o wide
[fedora@labkubedualhost-master-1 whereabouts]$ kubectl logs kube-multus-ds-fzdcr -n kube-system
Defaulted container "kube-multus" out of: kube-multus, install-multus-binary (init)
Error from server (BadRequest): container "kube-multus" in pod "kube-multus-ds-fzdcr" is waiting to start: PodInitializing

Seems I can make this happen anytime I ungracefully restart a node, worker or master it creates this error and stops pod network sandbox recreation completely on that node.

The fix mentioned above does work, but this likely means a power outage of a node will require manual intervention whereas otherwise without multus this is not required, this error should be handled properly.

+1. This seems like a pretty serious issue. Can we get a fix merge for it soon please?

Additionally can confirm this behavior. as @dougbtv mentioned... removing /opt/cni/bin/multus-shim works as a workaround.

+1 happend to me as well, cluster did not come up. Any chance to fix this soon?

same here, cluster kubespray 1.29

Certainly need to fix right away.

@dougbtv : Hit exactly the same issue. It helps by deleting /opt/cni/bin/multus-shim.

when could this be fixed?

Hit the same issue with kube-ovn. Already posted it there (kubeovn/kube-ovn#4470)
Only happen when I force delete the kube-ovn pod.

Also hit me today on a node that crashed.

Any indicator this fix is going to be picked up any time soon?

Had the same problem today, having a Talos kubernetes cluster. I modified the kube-multus-ds init containers to check for existing multus-shim file

Original command

command:
  - cp
  - /usr/src/multus-cni/bin/multus-shim
  - /host/opt/cni/bin/multus-shim

New command

command:
 - sh
 - -c
 - |
   if [ ! -f /host/opt/cni/bin/multus-shim ]; then
     cp /usr/src/multus-cni/bin/multus-shim /host/opt/cni/bin/multus-shim;
   fi

This worked for me 👍

thanks @iSenne already use this code in my kubernetes cluster.
hopefuly it will fixed the issue

Had the same problem today, having a Talos kubernetes cluster. I modified the kube-multus-ds init containers to check for existing multus-shim file

Original command

command:

  - cp

  - /usr/src/multus-cni/bin/multus-shim

  - /host/opt/cni/bin/multus-shim

New command

command:

 - sh

 - -c

 - |

   if [ ! -f /host/opt/cni/bin/multus-shim ]; then

     cp /usr/src/multus-cni/bin/multus-shim /host/opt/cni/bin/multus-shim;

   fi

This worked for me 👍

cp -f is arguably more correct.

Upgrading multus would mean you have an old shim file if you check for existence

Hey we are also really blocked by this issue. What can we do to push this forward?

An immediate mitigation that will get Multus running temporarily is to edit the DaemonSet directly and modify the cp command to add -f.

kubectl edit DaemonSet/multus -n <namespace>

scroll to the multus-installer initContainer
edit the cp command and add -f
cycle the pods

I think the concern at this point for folks wanting to use multus, is not about having a "workaround", but the seeming inability to get a fix up-streamed, leading to questions about the health of the multus project.

#1213 for example, has been open since Jan 18, and hasn't gotten any comment since Aug 12.

Please don't see this comment as knocking the devs hard work. It is very much appreciated, really. Just trying to gauge the health of the project though.

So crazy this has been ignored by maintainers this long. 🙄

FYI: I made a PR to add the -f flag to the angelnu/multus chart

This has been bothering me for quite some time, whenever I do node maintenance the whole cluster does not come up and I have to

# on each node
rm /opt/cni/bin/multus-shim

# afterwards delete the multus pods so they get redeployed
k delete -n kube-system pod/kube-multus-ds-*