The Prometheus receiver outputs unstable attributes when using k8s sd config, causing the cumulative to delta processor to fail to calculate delta values properly.

[h0cheung]

Component(s)

receiver/prometheus, processor/cumulativetodelta

What happened?

Description

Sometimes, attributes output are unstable:

• When a Pod's OwnerReference is modified, and we continue to collect metrics for that Pod at this point, some resource attributes in the output, such as k8s.replicaset.name, will change.
• Labels of a pod may change while running. If we add k8s labels by relabel in Prometheus's config, the data point attributes will be changed.

As long as some attributes are changed, the cumulative to delta processor will not calculate the delta value correctly.

Steps to Reproduce

1. Start a Pod which provides metrics via Prometheus endpoint.
2. Collect the metrics, and convert them to delta.
3. Edit the OwnerReference of it.
   3 (Alternative). Edit some k8s labels, which are added to the attributes by Prometheus relabel configuration.
4. Check the result of converting.

Expected Result

It should calculate the delta values correctly.

Actual Result

It cannot calculate the delta values. Because of the default behavior for start_timestamp, original values are preserved as delta.

Collector version

v0.118.0

Environment information

Environment

OS: Alpine Linux
Compiler(if manually compiled): (e.g., "go 14.2")

OpenTelemetry Collector configuration

Log output

Additional context

I think the best way is to calculate the delta values before adding any extra information. This means to make cumulative to delta processor embedded into prometheus receiver, which is not too easy.

Alternatively, I think we can add an option in cumulative to delta processor, to exclude some attributes. Then users should exclude those attributes that may change when running, to avoid this issue.

[github-actions]

Pinging code owners:

• receiver/prometheus: @Aneurysm9 @dashpole
• processor/cumulativetodelta: @TylerHelmuth

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

I think adding the cumulative to delta processor to the prometheus receiver isn't feasible.

Long-term, this should be improved by entities, which splits resource attributes into identifying and non-identifying. The cumulative-to-delta processor should be able to calculate deltas based on identifying attributes, and ignore non-identifying attributes.

I'll let @TylerHelmuth comment on the feasibility of excluding attributes.