fix(scheduler): Scheduler wait on server connect

[sakoush]

What this PR does / why we need it:

This change introduces a Synchroniser on the scheduler side to coordinate the startup process and especially to attempt to wait for expected model servers to connect before proceeding with any control plane operations. This is important in the case that the scheduler restarts so that we do no interrupt ongoing dataplane operations that should be valid to be served.

There are 2 implementations for this Synchroniser:

• SimpleSynchroniser: a timeout based sync for non k8s deployments
• ServerBasedSynchroniser: this is for k8s deployments that tries to match expected server states (from etcd k8s controller) with actual agents connecting to the scheduler. More description provided here.

Other changes in this PR:

• Explicitly mark the first ServerNotify event from the controller to indicate the list of expected servers collected from k8s etcd.
• Add ServerEvent in event hub so that the system propagates events related to servers, which are then consumed by the Synchronise. Thanks to @lc525 for this change.
• More test coverage for related part of the code base.
• Added param for helm to specify this value.

This PR builds on top of #5893.

Which issue(s) this PR fixes:

Fixes #
INFRA-747 (internal)

[sakoush]

The idea of the test with a pipeline is to make sure that we dont have any dropped messages.
Steps:

• Run k6 load
• Restart scheduler while k6 load is going

on 2.8.3 we have 15 messages dropped

with this change we dont have any dropped messages

[lc525]

lgtm, with some really minor comments. I think this addition is quite valuable in making the scheduler behave a lot better on restarts!