Scaling whois workers

[gl]

Hello, I have a use case using traceroute -A, massively. I don't want to discuss whether it's a good idea or not, but anyway, that's how it is ;)

For context, traceroute -A is basically dropping the ip address on port 43, and then reading the origin: attribute of the most specific route: or route6: returned. I'm sending probably ~1k reqs/s as an order of magnitude.

Setting RA_SERVER env variable on client side I'm pointing the traceroute to an in-house managed irrd 4.4.4 that is mirroring most of the usual sources with rpki resolution, and I'd like to scale the workers (up to 200 ~) to be able to answer this amount of whois requests from all those traceroute IP<->ASN queries, some coming from the other end of the world, with heavy latency. It breaks in various ways (pubsub issues, server restarting alone, it depends). To troubleshoot I've reduced the amount of workers and I'm limiting each requesting IP to 1 connection at a time with iptables.

I noticed various things:

• all whois workers download separately a copy of stuff from redis, all at the same time. This requires significant resources on the redis side, and I'm not sure that is the intended behaviour. Shouldn't it be downloaded once and then shared ?
• even once this is done, there is still a huge amount of queries going to the pgsql database, looking for rpsl_objects corresponding to the ip whois'd, is the redis data used at all ?

I'd be happy to have some guidance here :)

[gl]

Here we see 2Gb/s transferred from the redis server over ens4, with all the irrd-whois-server-listener active at once

[gl]

Ok I read a bit irrd code and I get why the pgsql part is normal; actually the redis is used only for sets, not rpsl entries.
I don't know why but I expected it to be also used as a query cache, but it's not the case obviously.

[mxsasha]

Yes, this is not something I've designed for. The Redis/memory buffer is specifically for AS->prefix and the members of all as-sets, as those are very frequent queries. The workers answer this from memory for minimum latency, Redis is used as an intermediate storage - collating this data from the SQL db is fairly expensive, so we only do it once. Also, this is only used for !a/!i/!g/!6, as we don't have the full RPSL object.

PostgreSQL query caching may be able to help, though then you still have the work of creating and running the query. You could try running with readonly_standby to prevent writes, to allow the query cache to remain valid. You'll need to tune the PostgreSQL settings yourself. If there is no process writing to the DB, the Redis/memory buffer should not need regular updates either, you could manually introduce some time variation to prevent the workers hammering Redis all at once.

None of this is really supported though. Of course, it's possible to expand IRRD to support this scenario, but that is a more significant effort.

[gl]

thanks for your answer !

My investigations led me indeed to the conclusion that I was actually killing postgresql, and that it would require a huge cluster to support the load.

I ended up spending a day coding a small whois proxy in C++ that is now deployed on all local traceroute probes and is caching (using libcappuccino) the (route:/route6:, origin) sets in memory.

traceroutes are reasonably stable, and the route objects barely change, especially for infra IPs, so it didn't make sense to always request the same info to the pgsql db. I decided for a LRU Cache with a Uniform TTL of 1 week, arbitrarily.

my traceroute command now looks like: "RA_SERVER=127.0.0.1 RA_SERVICE=4343 traceroute -A ...", and the whois proxy has the IP of irrd as an upstream server.

[gl]

btw, this question ends up being unrelated, but why do you download the redis data once per worker; can't you download it just once and share it across all of them ?

[mxsasha]

The reason is latency: each worker keeps AS->prefix and set->members (per source) in memory, and requesting this data from Redis is prohibitively slow. An alternative would be proper shared memory between the workers, but at the time we felt the added complexity was worse than the increased memory use. #338 has a lot of background