Add option to VMAgent to disable external label "vmagent"

[xiaozongyang]

Background

We deployed two groups of VMAgent which scrape completely same targets and send samples to two different vmstorage clusters for HA purpose. And there is a vmselect cluster querying these two vmstorage clusters simultaneously.

Suppose the two VMAgents are named vmagent-0 and vmagent-1 in same namespace vm. The raw sample from original target foo will be foo{vmagent="vm/vmagent-0"} in vmstorage cluster 0 and foo{vmagent="vm/vmagent-1"} in vmstorage cluster 1. When I search vmselect with query foo I got those two timeseries which caused duplicate data.

The motivation of this issue is to make the VMagent operator don't add external label vmagent so that I can make the samples in two vmstorage clusters are completely same. These two vmstorage clusters are completely equivalent.

Changes

1. Add new field enableVmagentExternalLabel to VMAgentSpec, its default values is true
   2 when the field is set to false the opeator don't add the external label vmagent

[xiaozongyang]

ping @f41gh7

Could you please help evaluate this requirement? I would be very happy to participate in the development of this feature.

[Haleygo]

As I know, we don't add external label vmagent: xx by default, but prometheus: ns/vmagentName.

operator/internal/controller/operator/factory/vmagent/vmagent_scrapeconfig.go

Lines 1377 to 1387 in ac9b6e6

	// Use "prometheus" external label name by default if field is missing.
	// in case of migration from prometheus to vmagent, it helps to have same labels
	// Do not add external label if field is set to empty string.
	prometheusExternalLabelName := "prometheus"
	if p.Spec.VMAgentExternalLabelName != nil {
	if *p.Spec.VMAgentExternalLabelName != "" {
	prometheusExternalLabelName = *p.Spec.VMAgentExternalLabelName
	} else {
	prometheusExternalLabelName = ""
	}
	}

Have you specified vmAgentExternalLabelName: vmagent?

It can be disable by:

spec:
  vmAgentExternalLabelName: ""