Retry running a CRD Manager if it fails #2202
Conversation
| level.Error(c.opts.Logger).Log("msg", "error running crd manager", "err", err) | ||
| errChan <- err | ||
| //TODO: What is a good default value for this? | ||
| retryInterval = 30 * time.Second |
There was a problem hiding this comment.
Let's use backoff to avoid thundering herd problems. You can use a similar approach to this: https://github.com/grafana/alloy/blob/main/internal/runtime/internal/controller/loader.go#L732
| for { | ||
| select { | ||
| case <-innerCtx.Done(): | ||
| //TODO: Log that the manager is exiting? |
There was a problem hiding this comment.
I'd say let's add a debug level just in case.
|
This PR has not had any activity in the past 30 days, so the |
PR Description
Setting up a k8s informer may time out (see #2161). In that case, we'd only retry setting up an informer the next time Alloy's config changes. We should retry sooner than that, since the config may never change.
Notes to the Reviewer
A few reasons why the PR is in draft stage:
PR Checklist