High CPU usage due to large number of VMRule

[lujiajing1126]

I am using VM Operator v0.53.0, and we have a single VMAlert and about ~2k VMRule selected by this instance.

We are now facing high CPU usage,

We investigate the issue with pprof,

The hot code path is shown above which may be introduced in 9d91c22. As I understand we have to reconcile (select) all VMAlert and then all selected VMRules (~ 2k in our case) for every single VMRule.

[f41gh7]

It looks like, most of the CPU time is spent on VMRule.Validate function call -

operator/api/operator/v1beta1/vmrule_webhook.go

Line 53 in 8b98d8e

func (r *VMRule) Validate() error {

It could by-passed, if VMRule has annotation:

kind: VMRule
metadata:
 annotations:
   "operator.victoriametrics.com/skip-validation": "true"

Operator tries to generate valid configuration file for vmalert and skips invalid VMRule objects.

Maybe, it's better to skip all runtime syntax checks and enforce it only with Validation webhooks. Or at least make it optional.

[ebensom]

@lujiajing1126 just another idea to mitigate: you may introduce VMRule sharding by deploying multiple vmalerts having selectors for distinct set of VMRules. For example, you may introduce 4 vmalerts, each having different selectors for rulesets:

ruleset: alerts-1
ruleset: alerts-2
ruleset: alerts-3
ruleset: alerts-4

And then you would apply an uniform (or random) distribution of the possible labels across your 2000+ VMRules.