Failed to reconcile etcd plane: Failed to add etcd member [etcd-server-k8s-2] to etcd cluster

[david-nano]

RKE version:

INFO[0000] Running RKE version: v1.5.10                 
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.17", GitCommit:"22a9682c8fe855c321be75c5faacde343f909b04", GitTreeState:"clean", BuildDate:"2023-08-23T23:37:25Z", GoVersion:"go1.20.7", Compiler:"gc", Platform:"linux/amd64"}

Docker version: (docker version,docker info preferred)
24.0.7

Operating system and kernel: (cat /etc/os-release, uname -r preferred)
Ubuntu 20.04

Type/provider of hosts: (VirtualBox/Bare-metal/AWS/GCE/DO)
BareMetal / VM's over nutanix

cluster.yml file: (Only masters in here)

# If you intended to deploy Kubernetes in an air-gapped environment,
# please consult the documentation on how to configure custom RKE images.
kubernetes_version: v1.24.17-rancher1-1
upgrade_strategy:
  max_unavailable_worker: 10%
  max_unavailable_controlplane: 1
nodes:
- address: 10.200.10.123
  port: "33"
  role:
  - controlplane
  - etcd
  hostname_override: server-k8s-1
  user: dcdevops
  docker_socket: /var/run/docker.sock
  ssh_key_path: ~/.ssh/id_rsa
  labels: {}
  taints: []
- address: 10.200.10.124
  port: "33"
  role:
  - controlplane
  - etcd #Just added
  hostname_override: server-k8s-2
  user: dcdevops
  docker_socket: /var/run/docker.sock
  ssh_key_path: ~/.ssh/id_rsa
  labels: {}
  taints: []
- address: 10.200.10.125
  port: "33"
  role:
  - controlplane
  - etcd
  hostname_override: server-k8s-3
  user: dcdevops
  docker_socket: /var/run/docker.sock
  ssh_key_path: ~/.ssh/id_rsa
  labels: {}
  taints: []

Steps to Reproduce:
Install cluster without etcd for master 2, and adding etcd for master 2 later.

Results:
Last Log:

INFO[0055] [reconcile] Check etcd hosts to be deleted   
INFO[0055] [reconcile] Check etcd hosts to be added     
INFO[0055] [add/etcd] Adding member [etcd-server-k8s-2] to etcd cluster 
{"level":"warn","ts":"2024-06-18T14:11:16.959325+0300","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00028c1c0/10.200.10.123:2379","attempt":0,"error":"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members"}
{"level":"warn","ts":"2024-06-18T14:11:17.15525+0300","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000749180/10.200.10.125:2379","attempt":0,"error":"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members"}
FATA[0056] Failed to reconcile etcd plane: Failed to add etcd member [etcd-server-k8s-2] to etcd cluster 

Not sure where to start debug and how to add new etcd node.

[github-actions]

This repository uses an automated workflow to automatically label issues which have not had any activity (commit/comment/label) for 60 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the workflow can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the workflow will automatically close the issue in 14 days. Thank you for your contributions.