Proposal: adding cmd/pid labels in order to distinguish same metric exported by multi-process

[hzzb]

In many situations, we often need to:

1. deploy several closely related service on one machine.
2. fork out child process as worker like nginx.

Basic metrics has no determined labels to identify which process exported them, they all look like this:

process_open_fds 9
go_goroutines 7

what's we really want is:

process_open_fds{cmd="bash", pid="10023"} 9
go_goroutines{cmd="bash", pid="10023"} 7

Its feasible for users to add custom labels to every metric, but that would be a lot of redundant code. so making these two labels builtin really helps.

How do you guys think this? I would like to make it works with a minor patch.

[prymitive]

This would basically mean that process_open_fds becomes an ephemeral metric, with a new time series generated each time a process restarts.
It would create problems for anyone who doesn't use multi-process setup, for example when you restart a service you'll end up with a new time series created for process_open_fds while the old one would be still present until it's declared as stale by Prometheus (5m by default), so for the first 5m after restart you have 2 process_open_fds metrics and you need to know which one is the new.

[bwplotka]

Thanks for this proposal!

I think we have a clear answer here because there is a simple rule:

If the labels are to identify the target (target metadata): Appended on your service discovery on scrape time.

The reason is that at the scraping time you can agree on one naming convention, one way of representing PIDs etc. You can survive a case when an application has this PID, but another not, because of the old version of client_golang.

IMO the cmd name and PID are such target metadata. They describe the SOURCE of data, not actually metadata about the categories of data).

Does this help @hzzb - is this something you can do with your service discovery? (:

[beorn7]

@hzzb how do you scrape the metrics from the various forked processes? I mean, they have to bind to a different port, each. And thus, they will get a different instance label attached by the Prometheus server. Your problem seems to be solved already.

@bwplotka how about turning this into a discussion, now that we have them?

[hzzb]

Ahaha. Thanks for reminding and suggesting!

More details about my situation. One executable starts up and forks out several subprocesses. They listen on same ip:port to serve incoming http traffic. A special http handler bound to /metrics is also installed on this ip:port.

When /metrics being scraped, the special http handler not only needs to collect the metrics of particular responding subprocess, but also metrics of all other subprocess. How does one subprocess read metrics of another subprocess? This could be done by some IPC mechanism such as shared memory.

This scenario needs some subprocess-specific labels to distinguish metrics between subprocesses. Of course, these extra labels could be appended in the special http handler. But I'm wondering if this could be a builtin. A switch flag may be needed too to switch off this builtin for non-multiprocess users.

[beorn7]

The one subprocess that serves the metrics could collect the metrics from the other sub processes in separate custom collectors or even registries. You can then use building blocks like https://pkg.go.dev/github.com/prometheus/[email protected]/prometheus#WrapRegistererWith and https://pkg.go.dev/github.com/prometheus/[email protected]/prometheus#Gatherers . See also
https://pkg.go.dev/github.com/prometheus/[email protected]/prometheus#example-Collector as an example.

Feel free to plug together a solution that works for you but I don't think prometheus/client_golang should have built-in first-class support for multi-process scenarios. Those are IMHO extremely rare in Go (where you usually use coroutines), and forking a Go program is not for the faint of heart anyway.