Ability to onboard custom nodes?

[akshatmittal]

I have some custom nodes that I'd like to onboard to the cluster. Would like to onboard them to the cluster so that specific pods can be scheduled on them.

Something that is/would be possible at some point?

[a-voitov-mitgo]

Hi! I’ve worked on a similar task. The idea was to launch a node in AWS and use it to create a tunnel into the AWS VPC. I managed to add my custom node to the cluster. The only thing that didn’t work was setting up full network communication between Rackspace nodes and my node. However, it was still possible to run certain pods on it.

The first step was to copy all the configuration from a Rackspace node to my own node — kubelet config, containerd config, etc.

Then, I created something like a service account in the cluster:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubelet
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubelet-node-role-binding
subjects:
- kind: ServiceAccount
  name: kubelet
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: system:node
  apiGroup: rbac.authorization.k8s.io

And then generated a token:

kubectl create token kubelet --duration=24h -n kube-system

After that, I used this token in the bootstrap config for kubelet.

I also created a kind: Node object in the cluster and attached a finalizer to it to prevent Rackspace from deleting it.

Then I started the kubelet, and finally had to approve the node CSR:

kubectl get csr
kubectl certificate approve ...

After that, the node started working.

[akshatmittal]

The only thing that didn’t work was setting up full network communication between Rackspace nodes and my node.

This is a deal breaker, for me at least. The resources in the Spot cluster need to be able to access services on the custom nodes.

[a-voitov-mitgo]

Hi @akshatmittal . I tried once again to connect the AWS node to the Rackspace cluster, and it worked. The intra-cluster network is functioning. In order for it to work, you need to use kubeProxyReplacement in Cilium and have the tunnel enabled.
You can use the instructions and configuration from here to set up the network.

Additionally, you may need to add a setting in the Cilium values.yaml, since the network interface names in Rackspace and AWS might differ:

devices: "eth0 ens5"

The only thing that didn’t work is the CSI driver.
Also, it's not possible to run kubectl exec into pods on this node because its external IP is being detected incorrectly.

[akshatmittal]

Okay that's awesome, will play around with this!

[supreeth90]

@akshatmittal Are you looking to import nodes from other hyperscalers like AWS to Spot control-plane ? Or add spot nodes to EKS controlplane ? We are working on ways to do the latter(add spot nodes to EKS controlplane).

[akshatmittal]

I'm trying to import on-prem nodes to the Spot control-plane (HA).

Adding to that, we also have some services in AWS I'd like to access inside the cluster, was thinking of importing an EC2 instance onto Spot as well.

Adding Spot nodes to EKS would certainly be interesting as well, but not what I'm looking for here.