Rework MSE query throttling to take into account estimated number of threads used by a query

[yashmayya]

• Add cluster configuration to allow limiting the number of multi-stage queries running concurrently #14574 introduced a mechanism to throttle multi-stage engine queries at the broker level using a new beta cluster config. The mechanism treated all queries as equivalent and assumed that queries were evenly distributed across brokers.
• This patch updates the mechanism to take into account the estimated number of threads that would be spawned in the servers for a query instead. The beta cluster config is also changed to reflect this. This is a better model since users no longer need to tweak the config based on their exact query workload and can instead use an estimated value based on the cluster size and instance sizes. Furthermore, mixed workloads will be better supported with larger queries potentially being blocked for longer while smaller queries are executed (query starvation and timeout is a possibility with this primitive model though).
• Note that prior to this change, there was actually an issue with the way the throttling threshold was determined - each broker calculated the threshold as maxConcurrentQueries * numServers / numBrokers. Since the max concurrent queries is a "per server" config, the calculation incorrectly makes the assumption that queries are executed on a single server (instead of assuming that they're dispatched to all servers which is a better assumption).
• With the change here to throttle based on the estimated number of threads, the throttling threshold becomes maxServerQueryThreads * numServers / numBrokers which only makes the assumption that queries are evenly distributed across brokers (and no assumptions about the fanout to servers). This feature isn't intended to completely prevent large queries from executing so the cluster config should be set to a value that is at least large enough to accommodate queries that can spawn up to maxServerQueryThreads * numServers / numBrokers number of threads.
• An alternate implementation could be to track the estimated number of threads on a per server basis (using the worker <-> server mapping in the query plan) but this has a lot more edge cases and also much higher overhead on large clusters with a large number of servers (since we'd need to acquire permits for many servers for every single query).

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 52.08333% with 23 lines in your changes missing coverage. Please review.

Project coverage is 63.77%. Comparing base (59551e4) to head (deaca57).
Report is 1598 commits behind head on master.

Files with missing lines	Patch %	Lines
...ot/query/planner/physical/DispatchableSubPlan.java	0.00%	15 Missing ⚠️
...requesthandler/MultiStageBrokerRequestHandler.java	0.00%	3 Missing ⚠️
...roker/requesthandler/MultiStageQueryThrottler.java	85.71%	0 Missing and 3 partials ⚠️
.../pinot/common/concurrency/AdjustableSemaphore.java	77.77%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #14847      +/-   ##
============================================
+ Coverage     61.75%   63.77%   +2.02%     
- Complexity      207     1613    +1406     
============================================
  Files          2436     2708     +272     
  Lines        133233   151259   +18026     
  Branches      20636    23349    +2713     
============================================
+ Hits          82274    96465   +14191     
- Misses        44911    47548    +2637     
- Partials       6048     7246    +1198     

Flag	Coverage Δ
custom-integration1	100.00% <ø> (+99.99%)	⬆️
integration	100.00% <ø> (+99.99%)	⬆️
integration1	100.00% <ø> (+99.99%)	⬆️
integration2	0.00% <ø> (ø)
java-11	63.74% <52.08%> (+2.03%)	⬆️
java-21	63.64% <52.08%> (+2.01%)	⬆️
skip-bytebuffers-false	63.76% <52.08%> (+2.01%)	⬆️
skip-bytebuffers-true	63.60% <52.08%> (+35.87%)	⬆️
temurin	63.77% <52.08%> (+2.02%)	⬆️
unittests	63.77% <52.08%> (+2.02%)	⬆️
unittests1	56.31% <0.00%> (+9.42%)	⬆️
unittests2	34.08% <52.08%> (+6.35%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.