Implement etcd member management in pre-terminate hook #435
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Currently testing. Getting errors in etcd after scaling down: ETCD cluster is not accessible, and API server is not recovering because of that |
Danil-Grigorev
force-pushed
the
reconcile-etcd-memebers-on-pre-delete
branch
3 times, most recently
from
ef72b5a
to
452143d
Compare
|
It seems the previous problem is fixed, but another problem appears on removal of the last machine. There is a discussion upstream about a potentially related issue https://kubernetes.slack.com/archives/C8TSNPY4T/p1725952675583209 |
Looks like a potential fix upstream will be available to use with the new v1.8.3 CAPI patch release scheduled for today, however we have not yet bumped CAPI to v1.8.x series in CAPRKE2 |
Danil-Grigorev
force-pushed
the
reconcile-etcd-memebers-on-pre-delete
branch
3 times, most recently
from
47e23e5
to
2ed8d90
Compare
Danil-Grigorev
force-pushed
the
reconcile-etcd-memebers-on-pre-delete
branch
3 times, most recently
from
6788009
to
5257503
Compare
|
Watch logs and metric collection causes failures in CI |
Danil-Grigorev
force-pushed
the
reconcile-etcd-memebers-on-pre-delete
branch
15 times, most recently
from
a547119
to
9f985d0
Compare
|
This PR depends on #440 |
tmmorin
reviewed
Sep 13, 2024
|
|
||
| var errs []error | ||
|
|
||
| for i := range machinesToDelete { | ||
| m := machinesToDelete[i] | ||
| logger := logger.WithValues("machine", m) | ||
|
|
||
| // During RKE2CP deletion we don't care about forwarding etcd leadership or removing etcd members. | ||
| // So we are removing the pre-terminate hook. | ||
| // This is important because when deleting KCP we will delete all members of etcd and it's not possible |
There was a problem hiding this comment.
Suggested change
| // This is important because when deleting KCP we will delete all members of etcd and it's not possible | |
| // This is important because when deleting RKE2CP we will delete all members of etcd and it's not possible |
tmmorin
reviewed
Sep 13, 2024
| } | ||
|
|
||
| // Note: Removing the etcd member will lead to the etcd and the kube-apiserver Pod on the Machine shutting down. | ||
| // If ControlPlaneKubeletLocalMode is used, the kubelet is communicating with the local apiserver and thus now |
There was a problem hiding this comment.
Yes, this comment can be simplified.
Signed-off-by: Danil-Grigorev <[email protected]>
Signed-off-by: Danil-Grigorev <[email protected]>
- Lint fixes Signed-off-by: Danil-Grigorev <[email protected]>
When the last machine is in a deleting state, this means that cluster is removed also. In such scenario, waiting for draining is not feasible, because it is performes only when node deletion is allowed. Which is not, due to cluster removal. Cluster API prevents draining with the "cluster is being deleted" error. Signed-off-by: Danil-Grigorev <[email protected]>
Signed-off-by: Danil-Grigorev <[email protected]>
Danil-Grigorev
force-pushed
the
reconcile-etcd-memebers-on-pre-delete
branch
from
9f985d0
to
fc6f21d
Compare
alexander-demicev
approved these changes
Sep 16, 2024
furkatgofurov7
approved these changes
Sep 16, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Which issue(s) this PR fixes (optional, in
fixes #<issue number>(, fixes #<issue_number>, ...)format, will close the issue(s) when PR gets merged):Fixes #431
Special notes for your reviewer:
Checklist: