Possible bug in ServiceMonitor → VMServiceScrape conversion (regex field incorrectly interpreted as array)

[rassvetalov]

We are experiencing an issue with the VictoriaMetrics Operator converting a Prometheus Operator ServiceMonitor resource into a VMServiceScrape. Our environment details:
• Kubernetes version: 1.29
• VictoriaMetrics Operator version: :v0.51.3
• Installation method: Helm

Description of the problem

We have a ServiceMonitor resource that includes metricRelabelings and relabelings with regex and replacement fields. When the VictoriaMetrics Operator attempts to convert this ServiceMonitor into a VMServiceScrape, we receive the following error in the operator logs:
{"level":"error","ts":"2025-01-14T19:43:42Z","logger":"controller.PrometheusConverter","msg":"cannot update VMServiceScrape","vmservicescrape":"karpenter","namespace":"kube-system","error":"VMServiceScrape.operator.victoriametrics.com \"karpenter\" is invalid: [spec.endpoints[0].relabelConfigs[0].regex: Invalid value: \"array\": spec.endpoints[0].relabelConfigs[0].regex in body must be of type string: \"array\", spec.endpoints[0].metricRelabelConfigs[0].regex: Invalid value: \"array\": spec.endpoints[0].metricRelabelConfigs[0].regex in body must be of type string: \"array\"]"}
It appears that the operator interprets the regex field as an array, even though in our YAML we have tried putting it as a string

Steps to reproduce
1. Install Prometheus Operator and VictoriaMetrics Operator in the same cluster.
2. Deploy the following ServiceMonitor (simplified example):

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: karpenter
  namespace: kube-system
spec:
  endpoints:
    - interval: 30s
      port: http-metrics
      path: /metrics
      metricRelabelings:
        - action: drop
          regex: ^karpenter_cloudprovider_instance.*
          sourceLabels:
            - __name__
        - action: replace
          replacement: 'itprod'
          targetLabel: team
      relabelings:
        - action: replace
          regex: ^(.*)$
          replacement: $1
          sourceLabels:
            - __meta_kubernetes_pod_node_name
          targetLabel: nodename
        - action: replace
          sourceLabels:
            - __meta_kubernetes_service_name
          targetLabel: app
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      app.kubernetes.io/instance: karpenter
      app.kubernetes.io/name: karpenter

Questions / Requests

• Could you please confirm if there is a known issue or bug with regex fields being interpreted as arrays during the automatic conversion from ServiceMonitor to VMServiceScrape?
• Are there any additional debugging steps or configuration we should apply to ensure the regex fields remain strings?
• Is there a known workaround that allows us to keep using ServiceMonitor specs with valid regex expressions without triggering this error?

[rassvetalov]

Update:
We tested the same ServiceMonitor on victoriametrics/operator:v0.40.0 and it worked without the regex array issue. This points to a potential regression introduced after v0.40.0.

[f41gh7]

Hello, did you try to update CRDs to the latest version? It may require to perform a manual CRD update due to helm limitations, see this doc https://github.com/VictoriaMetrics/helm-charts/blob/master/charts/victoria-metrics-k8s-stack/README.md#upgrade-guide

Operator defines if as preserve-unknown, it allows to have both array and string types.

              relabelConfigs:
                description: RelabelConfigs to apply to samples during service discovery.
                items:
                  description: |-
                    RelabelConfig allows dynamic rewriting of the label set
                    More info: https://docs.victoriametrics.com/#relabeling
                  properties:
                    action:
                      description: Action to perform based on regex matching. Default
                        is 'replace'
                      type: string
                    if:
                      description: 'If represents metricsQL match expression (or list
                        of expressions): ''{__name__=~"foo_.*"}'''
                      x-kubernetes-preserve-unknown-fields: true

[f41gh7]

Upcoming release of operator will contain a fix for this case. But currently, it's possible to update CRDs and resolve issue.

[rassvetalov]

Hello, I did not update any CRDs when switching between the old and new versions of the VictoriaMetrics Operator. On the older version, the exact same ServiceMonitor spec works correctly, but on the new version I’m getting an error about regex being interpreted as an array.
I’ve checked the CRDs for both ServiceMonitor and VMServiceScrape, and in both places regex is defined as a string So it’s unclear how updating CRDs alone would help.

[f41gh7]

Hello, I did not update any CRDs when switching between the old and new versions of the VictoriaMetrics Operator. On the older version, the exact same ServiceMonitor spec works correctly, but on the new version I’m getting an error about regex being interpreted as an array. I’ve checked the CRDs for both ServiceMonitor and VMServiceScrape, and in both places regex is defined as a string So it’s unclear how updating CRDs alone would help.

After v0.43.0 release definition of regex field was changed at VM CRDs, instead of string it accepts both string and []string. But it requires update of VM operator CRD. While it's not a breaking change for VM CRD objects, it's a breaking change for converted resources. It ll be addressed at the upcoming release as I mention above.

Related issue: #740

[f41gh7]

Issue was fixed at v0.52.0 release