Ingest storage local dev env: configure Mimir components to start after Kafka is healthy

[pracucci]

What this PR does

While working on #7142 and testing it locally using development/mimir-ingest-storage I frequently get mimir-write not starting successfully because Kafka is not ready after all the retries done by the Kafka reader (error: failed to fetch last committed offset). A simple solution looks configuring Mimir to wait until Kafka is ready, which is what I do in this PR (tested and fixes the problem).

Which issue(s) this PR fixes or relates to

N/A

Checklist

• Tests updated.
• Documentation added.
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX].
• about-versioning.md updated with experimental features.

[dimitarvdimitrov]

i used something like this locally https://stackoverflow.com/a/54824927/5683949, but if this version also works, LGTM

---

Also - the distributors and ingesters should retry and reconnect to a broker if it's unavailable, right? Do you know why they crash instead?

[pracucci]

the distributors and ingesters should retry and reconnect to a broker if it's unavailable, right? Do you know why they crash instead?

Ingesters fail the starting() if Kafka reader can't get the last committed offset after all the retries (and so the error: failed to fetch last committed offset).