Auto remove nodes in NotReady status?

[adyanth]

How long does it take to auto clean the nodes in NotReady status?

Context:
I might be misunderstanding it, but looks like the Cluster Autoscaler is not responsible for removing the node entry on scale down and k8s is supposed to do it. I am also unsure who the node controller is in this case, I am assuming it is part of k8s (and by extension, k3s).

[brandond]

Normally, removing terminated or deleted nodes is handled by the cloud controller, as it can query the cloud infrastructure provider's APIs to determine the actual state of the nodes. Because the default k3s stub cloud controller does not integrate into your infrastructure provider, it cannot know the actual state of your node, and will not take any action to remove nodes from the cluster.

The usual recommendation is to disable the stub cloud controller and deploy the one provided by your infrastructure provider.

[adyanth]

I see. I am writing a cluster autoscaler provider for Proxmox using k3s. I might just have to write the cloud controller too.

Thanks for the info! I couldn't find it anywhere.

[adyanth]

Couple of follow up questions. First, If I were to write my own, is there a way I can reuse functionality from the k3s stub controller? I see that most of them are unexported. Second, can I just disable and replace the NodeController or would I have to still provide all implementations for the cloud controller manager? Or better, can I run different cloud controllers for master nodes (k3s stub) vs worker nodes (my custom impl)?

Feel free to redirect me elsewhere if there is better information available I missed.

[brandond]

Is there a way I can reuse functionality from the k3s stub controller? I see that most of them are unexported.

Not easily, since the intended way to get an instance of the K3s cloud provider is via the kubernetes cloud-provider interface. You'd have to import the k3s cloud-provider into yours to make sure that the init function is called to register it, then call cloudprovider.GetCloudProvider("k3s", configReader) during your module's setup, and then just wrap the k3s cloud-provider's interface calls before or after doing whatever you want your logic provider to do.

Second, can I just disable and replace the NodeController or would I have to still provide all implementations for the cloud controller manager?

You have to implement the interfaces, but you can just have the ones you don't care about return cloudprovider.NotImplemented error and it should accept the no-op.