Multithread DNS cache warmups performed over UDP

[jeremyjpj0916]

Right now checking kong tcp dumps on startup, we have slow DNS startup warmup, as kong iteratively goes through each dns fqdn to resolve and does it 1 by 1 for 1000s of records in 90 seconds:

2024-07-09T15:15:35.742279049-05:00 2024/07/09 20:15:35 [notice] 49#0: *2133 [lua] warmup.lua:47: warming up DNS entries ..., context: ngx.timer

... giant delay...

2024-07-09T15:17:05.532245176-05:00 2024/07/09 20:17:05 [notice] 49#0: *2133 [lua] warmup.lua:81: finished warming up DNS entries' into the cache (in 89791ms), context: ngx.timer

Smarter implementation would be multithreading 100s of lookups at a time and reading the responses back in multithreaded to support large scale dns cache warmups so slower records or timeouts etc. don't hold up the caching. Just noting it here for an enhancement in Kong's future. Using kong 3.7.1 in this env.

[chronolaw]

@chobits , perhaps you will be interested on it.

[jeremyjpj0916]

You would think it doesn’t matter because dns is fairly fast but I have noticed in big corp land with private dcs there can be all kinda infra hops and fws to go through so dns latencies sometimes can avg 10-35ms to crazy times as long as 3-5 seconds on a super small % of records when you have name servers delegating to gtms or having to do further recursive lookups elsewhere. Now in some network zones almost all dns responses are around 1-5ms tops so typically its all pretty fast but multithreading it could yield better performance for all cases regardless.

[chobits]

It is indeed an issue when you have a large number of hosts to resolve. To address this, we need to reconsider the implementation of DNS warmup, such as introducing an option to disable it, or setting a time limit, or performing it in the background. This will take some time, so I'll file an internal ticket to track it.

Or like multithreads, or use mlcache to do one resolution for all workers

[chobits]

kag-4931

[jeremyjpj0916]

Will say that most resolve quickly but yeah some % in larger envs of hosts will just have slower times so if it parallel processed 10-100 at a time I think would see dramatic speed up from 90s back to 9s or less which is reasonable for larger gateway instances.

My ultimate goal for kong oss is to see it performant for up to 30k proxies/backends as that means it can meet 99.99% of api gateway use cases. Been getting better to be close to that over the years I have worked w it.