Blocky with redis

[larivierec]

Good morning,

Yesterday I installed 0.19 of blocky in my cluster, however, I noticed that grafana blocky chart had really terrible average response times > 1s.

I then reverted back to 0.18 and also restarted redis to be sure and these response times were back to normal ~50ms.

My question is should I have restarted redis when upgrading to 0.19? Would this have perhaps helped performance? Note I run in average of 2-5 replicas of blocky.

I could also create an issue if that could help.

I had to enable the filtering option on AAAA because the bootstrap was trying to resolve ipv6 and was timing out/failing for some of my file downloads.

I seem to get some broke pipes in the blocky logs
[2022-06-08 13:41:23] ERROR can't write message: write tcp 172.16.134.239:853->192.168.10.204:45980: write: broken pipe

these are the only things I have changed and for the time being have reverted.
larivierec/home-cluster@05d4077

[ThinkChaos]

I don't think restarting Redis would do much, but I'm not very familiar with that code.

Any chance you can collect trace log for a query that takes a long time so we can narrow this down?

For the bootstrap issue, I think it's not the resolving that's problematic (since resolving AAAA over IPv4 is not an issue), but the connecting to the file host over HTTP.
The code uses a random IP from what was resolved for the host. And, unless you filtered A or AAAA, it resolves both. Meaning by default blocky will sometimes try to connect via IPv6 to the file host.
Is IPv6 connectivity not working in your blocky host?

The same behavior is used for any DoH upstream. That code will retry the query twice (3 tries total), so it might (randomly) start with IPv6, timeout/fail, retry and choose IPv4 at some point. Basically wasting time trying IPv6.
This shouldn't happen if you filtered AAAA queries though since there'd be no IPv6 to randomly choose. If I understand correctly you had those filtered when you measured those times; is that right?

The broken pipe error makes it sound like the client is disconnecting before blocky responds.
Do your clients have a shorter timeout than blocky?

One thing relevant here is that current UpstreamResolver code uses the configured timeout as the timeout for a single try.
So each request from a blocky client can take as much as 3x the configured timeout + 100ms of delay in between each try (so 2 x 100ms).
You might want to lower your timeout config, but that doesn't solve your long average response time, unless you're getting a lot of timeouts.
@0xERR0R would you be open to changing how the timeout configuration works?
Maybe we should open an issue to discuss that, but I think expanding the config to allow changing the number of tries, and their delay, would help make this behavior clear to the user.

[0xERR0R]

Since the IP of the bootstrap is known, maybe we should assume, that we can always do A query if it is IPv4 and AAAA for IPv6?

[ThinkChaos]

So this would be the behavior?

⬇️bootstrap IPs, ➡️Connect using	v4	v6
None	✔️	✔️
v4	✔️
v6		✔️
Both	✔️	✔️

I think it's sensible default behavior but also think adding an option to explicit control IPv6 usage would be beneficial since more explicit.

[0xERR0R]

Yes, this could work. We could also provide a new config option "bootstrapNetwork" (or something more meaningful) as enum with ipv4, upv6 or both as values. Default is both.

[ThinkChaos]

I just realized GitHub doesn't notify a discussion when it has been mentioned, so doing it manually: PR #589 is linked.

[larivierec]

Thanks again for the feature, working like a charm 😃