Trigger leader election but exclude *this* node

[sebadob]

Hey,

I am really happy with how openraft works so far, you have done an awesome job! The only big thing left is the async heartbeats which will come with the next version.

I do have one issue though. In some situations, I am using openraft to create a raft with ephemeral in-memory nodes for caching only, which means they won't have their state saved after a restart. Everything works perfectly fine, except for one situation that I am trying to solve now.

When I am doing a rolling release of for instance 3 pods inside K8s, I sometimes end up with a "crashing" node that's getting stuck because of internal race conditions. These are not inside openraft itself, but probably inside my own logic wrapped around the whole thing. The problem comes up because nodes sometimes restart too fast, so that I end up in a state where the next node tries to join the cluster again when the leader election has not finished yet, but then the next-next node shuts down already, and so on.

I can "solve" this easily when I just add a timeout of for instance 10 seconds after calling raft.shutdown(), but this is not the cleanest and most robust solution. What I would like to do is to trigger a leader election, if the node I am shutting down currently is the leader, and then wait until the election has finished. This would make the whole process way more stable.

So, my question is:
When I am calling .shutdown(), it seems to me that the node is simply shutting down and when heartbeats time out, openraft will trigger a new election, am I right with this assumption? Or does openraft actually trigger an election before returning from .shutdown() to make the switch seamless?
If .shutdown() does not handle this automatically (would be a cool feature I guess), is there a way that I can exclude "this" node, when I manually raft.trigger().elect(), whithout changing the membership and cause possible hickups?

[schreter]

There is a new feature for transferring leader to another node in master.

However, for your use case, I think a much cleaner solution would be to add a 4th Pod and change the membership to include the new Pod and exclude the one you are shutting down. After the membership is changed, shut it down, then do the same on the other two Pods. This is also less risky for your in-memory data, since you always have 3 replicas running.

[sebadob]

There is a new feature for transferring leader to another node in master.

So this would probably come with openraft-0.10, right? Because that's exactly what I was looking for.
Unfortunately, for different reasons I can't start up a 4th node before shutting down another one.

It would not be the end of the world if I would lose cached data, because it can be rebuilt easily. A way bigger issue in my case is, if the raft get's stuck and the node will be in a non-working state for like a minute or something like that, until K8s kills and restarts it. I can also not set is to ready until it has joined the cluster or leader election has finished, because then I would get into trouble during a cold start of the cluster, since nodes start one after another.

But knowing that it comes with the next release helps a lot already. Then I will stick with the manual shutdown delay for now, which is not clean but works so far.

Thank you a lot!

[drmingdrmer]

Adding a 4th pod will trigger a full size data transmission from the leader to the new node. This may not finish soon enough. :(

[sebadob]

Yes, but that is pretty easy to solve. You can use the K8s scheduler to your advantage, as long as you handle ready checks properly.

For instance, if the pod has not joined any Raft cluster it should join (because maybe the other nodes have not been started yet), it will report ready for the first 60 seconds after application startup, even if it's not, to not get into a chicken and egg problem during first cluster initialization.

The first ready check from K8s will come after 3 * election_timeout_max. This means if the pod has been a cluster member already and is maybe just restarting, it will have joined the cluster until this time for sure. If this is the case within the first 60 seconds, I can check how much the current node is lagging behind in log_id. If The node is lagging too much, it will simply return not ready in the first 60 seconds and K8s will delay the shutdown of the next node, until the restarted one is flagged as ready. This is working pretty good so far.