The querier worker has not been configured...

[SunSparc]

I started Mimir and added it as a data source in Grafana.

URL: https://mimir.mydomain.com/prometheus
HTTP method: POST (also tried GET)
Prometheus type: Mimir
Mimir version: > 2.3.x
Save & test: "Data source is working"

Then I headed over to the Explore section of Grafana to test it out:

"Loading metrics..."
"Validation failed"
"(No metrics found)"

HTTP 400 - {"status":"error","error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}
HTTP 422 - {"status":"error","errorType":"execution","error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}

HYYP 500 - {
    "status":"error",
    "data":null,
    "errorType":"server_error",
    "error":"unable to retrieve rules from ruler 10.11.12.13:12345: rpc error: code = Canceled desc = grpc: the client connection is closing"}

So I opened the server logs and found this message near the top:

The querier worker has not been configured with either the query-frontend or query-scheduler address. 
        Because Mimir is running in monolithic mode, it's attempting an automatic worker configuration.
        If queries are unresponsive, consider explicitly configuring the query-frontend or query-scheduler
            address for querier worker.

Might this have something to do with Grafana not being able to pull metrics from Mimir?

And if so, I am confused. In the configuration reference, I cannot find any blocks/sections called querier worker or any variation of that. Perhaps it just refers to an instance of querier? But even if that is the section that is being reference, there is no field/parameter in that section called query-frontend or query-scheduler. The only place I can see a query-frontend is in the ruler section. I see a frontend_worker section. Is that the section being referred to? Because that actually has an address field...

(As a side note, is it just me, or is the naming of things in Mimir configs and docs inconsistent?)

[pracucci]

I see a frontend_worker section. Is that the section being referred to? Because that actually has an address field...

That's correct. The log message you got is about it. However, if you're running Mimir in monolithic mode and you have only 1 replica, then you don't need to configure it and the log is just telling you that's the worker has been automatically configured.

unable to retrieve rules from ruler 10.11.12.13:12345: rpc error: code = Canceled desc = grpc: the client connection is closing

This is not related to a request run by a query. This log is about fetching rules, that in Grafana would happen if you open the "Alerting" panel.

HTTP 422 - {"status":"error","errorType":"execution","error":"rpc error: code = Canceled desc = grpc: the client connection is closing"}

Can you try to manually run a query with curl against Mimir? The easiest way is to run an instant query using this API endpoint. It would help to narrow down the problem, excluding Grafana to begin with.

[charleskorn]

When you run the request directly on the server, are there any logs generated by any of the replicas that might indicate what the problem is?

[SunSparc]

These are the log entries that are generated at the time of request:

25:41.683101642Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=10.10.1.1:9096 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
25:41.68315335Z caller=pool.go:193 level=warn msg="removing ingester failing healthcheck" addr=10.10.1.2:9096 reason="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
25:41.683846675Z caller=retry.go:78 level=error user=anonymous msg="error processing request" try=4 err="rpc error: code = Code(500) desc = {\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: rpc error: code = Canceled desc = grpc: the client connection is closing\"}"
25:41.684039811Z caller=handler.go:250 level=info user=anonymous msg="query stats" component=query-frontend method=GET path=/prometheus/api/v1/query response_time=1m11.545174053s query_wall_time_seconds=0 fetched_series_count=0 fetched_chunk_bytes=0 fetched_chunks_count=0 fetched_index_bytes=0 sharded_queries=0 split_queries=0 param_query=asdf status=failed err="rpc error: code = Code(500) desc = {\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: rpc error: code = Canceled desc = grpc: the client connection is closing\"}"
25:41.684133408Z caller=logging.go:86 level=warn traceID=4e51f5f1822ac25c msg="GET /prometheus/api/v1/query?query=asdf (500) 1m11.545533315s Response: \"{\\\"status\\\":\\\"error\\\",\\\"errorType\\\":\\\"internal\\\",\\\"error\\\":\\\"expanding series: rpc error: code = Canceled desc = grpc: the client connection is closing\\\"}\" ws: false; Accept: */*; User-Agent: curl/7.81.0; "

[charleskorn]

That looks like either a networking issue between the replicas, or the replicas are crashing or not available. If you look at the ingester ring status page on each replica, does each one show all three instances as healthy?

[SunSparc]

Yes, it shows all three as ACTIVE. However, it appears that each of them is using an external IP address instead of the internal addresses that I specified in the config file in the server section. I do not see a way to specify an address for the ingesters. The problem with the addresses they are using is that those are inaccessible. This makes sense why they are failing the health checks. Where are the ingesters getting their addresses?

Thinking maybe the addresses were cached, I tried nuking everything and starting the cluster from scratch. Still have those addresses. Perhaps they are discovering the external addresses for themselves...

I had the thought look at the ring configurations. Behold, there it was: instance_interface_names ("List of network interface names to look up when finding the instance IP address."). I just had to pick the correct interface.

Wow. Thanks for the help!!

[SunSparc]

For future reference, I noticed different yet similar log entries about the ruler:

"removing ruler failing healthcheck"

Again the problem was that the external IP address had been chosen by default. Specifying the instance_interface_names parameter fixed that again.