improve unhealthy status management

[srene]

right now unhealthy status is set by sending an unhealthy event. this is an issue in case we want to automatically recover from unhealthy status and it is required to refactor the logic to allow that.

[omritoptix]

also adding, separating freezing and unhealthy as currently they are coupled. we need to map the various areas we currently freeze/emit unhealthy and according to that decide what each flow does

[mtsitrin]

AFAIU, the freeze state is for unrecoverable errors
It should happen on

• validation errors
• errors in block manager loops (production/syncing)
• role switch (proposer -> FN, FN->Proposer)

---

what use cases of recoverable unhealthy cases do we have?

[omritoptix]

@mtsitrin one example would be skew (which require pausing, but may recover after e.g DA/settlement is reachable)