OpenTelemetry support

[Gorbush]

Hi.

Is there any plan to support traces and metrics for OpenTelemetry?
https://opentelemetry.io/
We use HoneyComb for that and need to get the metrics for BazelRemote instances.
https://www.honeycomb.io/

Thank you!

[mostynb]

Hi.

Re metrics: bazel-remote currently supports prometheus. I'm not sure of the current status of opentelemetry metrics, is it still kind of immature?

Re tracing: I have been holding off adding this feature because I'm not sure how useful it would be. For bazel-remote installs that only use the disk cache, I suspect that disk iowait metrics are probably more useful than traces for individual requests. Maybe things are different for installs that use s3/gcs/azblob/http backends though?

[smocherla-brex]

Just to piggyback on this, having OTEL support would be especially helpful in tracing problems with requests to s3 or other proxy backends from the cache. Right now, it's quite difficult to debug issues when problems arise there as the access logs aren't always very helpful (from the minio SDK) and neither are metrics.

If you're open to it, I'm happy to open a PR that tries to setup that integration.

[mostynb]

@smocherla-brex: a PR would be welcome- if the diff starts to grow too large please discuss here first :)

[smocherla-brex]

@mostynb I have an initial PR here #873, from initial testing it seems to be working as expected and is strictly opt-in behind a config.