Full TCP sync in a large-scale cluster may "revive" a gone node for a long time

[vitalif]

Hi.

I'm testing memberlist in a ~4500 node cluster and I observed another nontrivial issue where a removed node was being "revived" and lost again many times by different nodes during ~6 hours after it was gone.

As I understand, it's related to full TCP sync mechanism, because normally "aliveNode" messages only originate from the node itself. BUT during full TCP sync, these "aliveNode" messages may originate from any node which participates in the full sync. Which may lead to "migratory" aliveNode messages in the cluster if it's large enough, especially when you restart the whole cluster at the same time. I.e. different nodes mark it as alive, re-broadcast the gossip message, then, at some time, they try to ping the node, discover that it's actually dead and remove it back. In my case it was happening for ~6 hours, until the node was finally removed from all nodes.

I haven't encountered this problem after I disabled full sync.