Simulate impact of shard movement using shard-level write load #131406
Conversation
nicktindall
added
>non-issue
:Distributed Coordination/Allocation
elasticsearchmachine
added
Team:Distributed Coordination
|
Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination) |
…ad_modeling_to_balancer
|
|
||
| public class WriteLoadPerShardSimulator { | ||
|
|
||
| private final ObjectFloatMap<String> writeLoadDeltas; |
There was a problem hiding this comment.
👍 I changed to simulatedWriteLoadDeltas, we only store the delta from the reported/original write load here. The idea there is that if no delta is present, we can just return the original NodeUsageStatsForThreadPools instance.
| } | ||
| } | ||
| writeShardsOnNode.forEach( | ||
| shardId -> writeLoadPerShard.computeIfAbsent(shardId, k -> new Average()).add(writeUtilisation / writeShardsOnNode.size()) |
There was a problem hiding this comment.
Do you equally divide write-load across all write shards on node?
There was a problem hiding this comment.
Yeah, this is just a stop-gap until we get actual shard loads, which should work as a drop-in replacement.
There was a problem hiding this comment.
Ok, I was thinking maybe we should have some heuristic from already available data. Otherwise signal/noise ratio is too high. It's not uncommon to have hundreds of shards, and estimation has little to no impact on a single shard.
For example use shardSize heuristic, the larger size more likely it would have write-load. Lets say linearly increase weight of those shards as size approaches 15GB. And then decrease weight as they approach to 30GB since we would roll-over them (most of the time) if size <15GB then size/15GB else max(0, 1-size/30GB)
There was a problem hiding this comment.
We'll have actual shard write loads shortly. Hopefully we can avoid all this guessing entirely.
…ad_modeling_to_balancer
|
I might hold off merging until we get #131496 merged, I think we can avoid fudging the shard write loads |
nicktindall
changed the title
I've been back and forth on this a bit, but I think going for something simple is best. When we start receiving shard write load estimates from the nodes that should be able to plug those in and this should "just work" (assuming I've understood shard-level write load correctly.We ignore queue latency in the modelling because I don't think we're going to look at it in the decider, and I can't see how we could estimate how it would change in response to shard movements (it's a function of the amount the node is overloaded AND how long it's been like that, and back-pressure should ideally keep a lid on it).