NodeConnectionManager.createConnectionMultiple should stagger connection attempts to reduce duplicate connections

[tegefaulkes]

Specification

createConnectionMultiple is used when we are attempting a direct connection and we have multiple possible addresses to attempt. It is equvalent to doing multiple createConnection calls concurrently with some clean up logic sprinkled on top.

It's possible that multiple addresses in the list is valid. In this case we have a race condition where multiple connections are established simultaneously. This is not actually a problem since we fully allow for this. However we want to cut down on duplicate connections if possible.

To this end we need to stagger the connection attempts by some short delay. This will give some grace time for single connections to connect before more are attempted concurrently. If the connections can be established within this delay then only a single connection should be made before we attempt the clean up step.

This should reduce the number of connections fully established concurrently.

I'd say add a test to demonstrate reducing the number of concurrent connections established. But this is a bit indeterminate so we can't really create a test condition for it. Even if it works, It's likely to fail on the CI.

Tasks

1. Add a staggerDelayTime parameter to the createConnectionMultiple function.
2. Add logic to wait the staggerDelayTime between starting each connection in the list.

[CMCDragonkai]

Should investigate if we can separate the RPC method dispatch from the NC creation.

So something like this:

withRPC(addr, (m) => {
  await m.f();
});

That should allow for the possibility of selecting the preferred NC whichever one it ends up being, while allowing concurrent NC attempts, and then enabling aggressive NC garbage collection using STONITH.

[CMCDragonkai]

I think the ability to route RPC calls to the preferred node connection will be especially useful for long-running handlers that may be making multiple RPC calls.

[CMCDragonkai]

It's not a critical problem to have multiple node connections, so this is primarily and optimisation and reduction of memory usage.