adding additional labels for an EXTERNAL probe

[jbkc85]

I am attempting to run an EXTERNAL probe that provides additional label values (based on the validation and HTTP examples). Here is what I came up with:

	payload = append(payload, fmt.Sprintf("service_resource_health map:service,database:%d,elasticsearch:%d",1,1)

service_resource_health=map:service,database:1,elasticsearch:1

Yet, cloudprober does not want to read this and quickly removes it from the parsed metric values. Am I doing something wrong here? Is adding additional labels on runtime of an EXTERNAL probe not allowed?

[manugarg]

Sorry for the lack of documentation on special data types -- map and distributions. What you're trying to use here is the 'map' data type.

Your printf statement and what you wrote afterwards don't match, but if your string looks like the following it should work:
service_resource_health map:service,database:1,elasticsearch:1

An example map string from the tests:

cloudprober/metrics/map_test.go

Line 84 in 361f0d9

expectedString := "map:lat,p50:20.000,p99:4000.000"

Can you confirm that's how your output looks like?

I tested the same at my end and it worked fine. I ran this script through the external prober:
#!/bin/bash

echo "service_resource_health map:service,database:1,elasticsearch:1"

and it generated following metrics for prometheus:
service_resource_health{ptype="external",probe="external_test",dst="dummy",service="database"} 1.000 1562695144376
service_resource_health{ptype="external",probe="external_test",dst="dummy",service="elasticsearch"} 1.000 1562695144376

[jbkc85]

$ ./releases/cloudprober-0.0.1 --backend myservice --address dev-myservice-01
service_health_latency_ms 560.506347
service_resource_health map:service,database:1,elasticsearch:1

Jul 09 18:12:12 cloudprober[13115]: cloudprober 1562688279673709478 1562695932 ptype=external,probe=myservice,dst=dev-myservice-01,environment=dev,service=myservice service_health_latency_ms=242.496

CFG looks like:

# Service: myservice
# Type: EXTERNAL
probe {
  type: EXTERNAL
    targets {
      host_names: "dev-myservice-01"
    }
  interval_msec: 60000
  timeout_msec: 10000
  name: "myservice"
  external_probe {
    mode: ONCE
    command: "/opt/cloudprober/bin/cloudprobers --backend myservice --address @target@"
    output_metrics_options {
      additional_labels: "environment=dev,service=myservice"
    }
  }
}

[jbkc85]

for some reason it just skips over the former external cloudprober output.

The golang code (sorry for the earlier pasting, i took out the = sign) looks like this:

	payload = append(payload, fmt.Sprintf("service_resource_health map:service,database:%d,elasticsearch:%d",1,1))
        return strings.Join(payload, "\n"), err

Which then is simply returned via fmt.Println of the returned string

[manugarg]

@jbkc85 Can you also share which cloudprober version are you using? Support for parsing maps was added to external probe in May 2019.. and is available only in the version from the head (or docker latest label).

[jbkc85]

# ./cloudprober --version
v0.10.2

[jbkc85]

that may explain it. I was seeing it done with the validator area, so I figured it would be available in the external probe area.

I dont suppose this will be in a 'release' anytime soon will it? 😬

[manugarg]

Argh, sorry :(

It will be. I'll cut a new release some time this week. Just waiting for last few changes to go in.

[jbkc85]

@manugarg thats awesome! I just built off of master and verified its working. I will be happy to submit some documentation too. We are basically transitioning all of our Nagios 'health' checks (which read JSON) to a interface-enabled golang external probe triggered via CloudProber. We also have consul-template feeding our cloudprober config, so though the cfg itself is super messy right now, it will automatically generate upon new deployments (we dont have K8s or Docker yet and still check server by server).

Anyway, will be happy to contribute whenever/wherever we can!

[manugarg]

@jbkc85 That's excellent. If you can write a Github gist or share in the form of a Github issue (especially here: #123) how you're using cloudprober that will be great. If it works for you, you can also update the documentation directly at https://github.com/google/cloudprober/tree/master/docs/content.

Also, just a heads up, I'll most likely cut a new release (v0.10.3) on Tuesday (2019-07-16) now.

[manugarg]

Just circling back here. v0.10.3 has been released in case you want to use the named version.

[MohamedMSaeed]

Hi @manugarg,

"service_resource_health map:service,database:1,elasticsearch:1"

I'm trying to use more that one label but it gives me an error. Can you tell me if this is possible?

If I printed this:

fmt.Printf("service_resource_health map:service,database:1\n")

it will work fine. And label service=database will be added. But what if I want to add another label called env=prod?

I was trying something like

fmt.Printf("service_resource_health map:service,database:1 map:env,prod:1\n")

Thanks

[manugarg]

@MohamedMSaeed No, I'm afraid, that won't work. Our map type supports only 1 key.

As a workaround, you can consider splitting your probe into two:

probe {
  name: "probe_that_in_prod"
  additional_label {
    key: "env"
    value: "prod"
  }
  external_probe {
   ..
  }
}


probe {
  name: "probe_that_in_staging"
  additional_label {
    key: "env"
    value: "staging"
  }
  external_probe {
   ..
  }
}

Additional label example:
https://github.com/google/cloudprober/blob/master/examples/additional_label/cloudprober.cfg

[MohamedMSaeed]

@manugarg

Thank you for your fast reply.

Can I use dynamic labels in config file and values comes from the go code that I run? If so it will solve my issue.