potential to miss FAILURE_NOTIFICATION when multiple instances killed at same time

[glassfishrobot]

Bug was uncovered during a code review. The bug is a FAILURE notification could
be missed when 2 or more more instances are killed at same time. (Note that
given the race condition between node agent restarting a killed instance and the
failure notification, only a test that kills the node agent and then kills
instances can be assured of seeing a FALIURE_NOTIFICATION for each server
instance killed. A node agent can restart a server instance before shoal
reports it as FAILED.)

HealthMonitor.InDoubtPeerDetector.processCacheUpdate() iterates over all
instances in cluster checking if any are in doubt. If one instance is detected
to be indoubt, HealthMonitor.InDoubtPeerDetector.determineInDoubtPeers() notifies
the FailureVerifier thread to process current cache looking for InDoubtPeers to
verify which instance should have FAILURE_NOTIFICATION sent.

synchronized (verifierLock)

{ verifierLock.notify(); LOG.log(Level.FINER, "Done Notifying FailureVerifier for " + entry.adv.getName()); }

The notification signal from InDoubtPeerDetector thread to FailureVerifier
thread is the weak link in this bug. When multiple failures happen at once, the
code is currently written to act on the first instance failure immediately. The
InPeerDoubtDetector should iterate over all instances AND if one OR more
instances are in doubt, then it should notify the FailureVerifier thread to run
over all instances in cluster cache.

Bug could be that InDoubtPeerDetector, runs twice, one notifiying
FailureVerifier() to run on instance cache and it detects first killed instance.
The second time the InDoubtPeerDetector runs, it could notify the
FailureDetector while it is still working on verifiying first failure (with a
snap shotted cache). The second notify to a running FailureVerifier thread will
have no impact and the FAILURE_NOTIFICATION for the second killed server
instance will be detected much later when the next failure occurs or the client
is shutdown.

Environment

Operating System: All
Platform: All

Affected Versions

[current]

[glassfishrobot]

@glassfishrobot Commented
Reported by @jfialli

[glassfishrobot]

@glassfishrobot Commented
This issue was imported from java.net JIRA SHOAL-74

[glassfishrobot]

• Issue Imported From: https://github.com/javaee/shoal/issues/74
• Original Issue Raised By:@glassfishrobot
• Original Issue Assigned To: @jfialli