Metric with different tag names not output properly in PrometheusMeterRegistry

[mlubavin-vg]

Hello, I am using Spring Boot 2.4.3 with micrometer 1.6.4

When I create a PrometheusMeterRegistry, and populate it with a timer with one set of dimensions, and then another timer with a different set of dimensions for the same metric name, the output from PrometheusMeterRegistry.scrape() only includes the first set of dimensions. It does not include the data for same operation name but different dimensions.

Here is a fully working test case:

package com.example;

import io.micrometer.prometheus.PrometheusConfig;
import io.micrometer.prometheus.PrometheusMeterRegistry;
import org.junit.jupiter.api.Test;

import java.util.concurrent.TimeUnit;

public class CommonMetricServiceTest {

    @Test
    public void testDifferentTagsForTimedMetric() {
        PrometheusConfig config = new PrometheusConfig() {
            @Override
            public String get(String key) {
                return null;
            }
        };

        PrometheusMeterRegistry meterRegistry = new PrometheusMeterRegistry(config);

        meterRegistry.timer("operationName", "tag1", "value1", "tag2", "value2").record(100, TimeUnit.MILLISECONDS);
        meterRegistry.timer("operationName", "tag1", "value1", "tag2", "value2DIFFERENT").record(100, TimeUnit.MILLISECONDS);
        meterRegistry.timer("operationName", "tagA", "valueA", "tagB", "valueB").record(100, TimeUnit.MILLISECONDS);
        
        System.out.println(meterRegistry.scrape());
    }
}

The output is:

# HELP operationName_seconds  
# TYPE operationName_seconds summary
operationName_seconds_count{tag1="value1",tag2="value2",} 1.0
operationName_seconds_sum{tag1="value1",tag2="value2",} 0.1
operationName_seconds_count{tag1="value1",tag2="value2DIFFERENT",} 1.0
operationName_seconds_sum{tag1="value1",tag2="value2DIFFERENT",} 0.1
# HELP operationName_seconds_max  
# TYPE operationName_seconds_max gauge
operationName_seconds_max{tag1="value1",tag2="value2",} 0.1
operationName_seconds_max{tag1="value1",tag2="value2DIFFERENT",} 0.1

The expected output should also contain metrics for operationName with tagA="valueA",tagB="valueB"

Note that calling meterRegistry.getMeters() does correctly return the 3 different PrometheusTimer objects.

[checketts]

Prometheus is rejecting the mismatched tagA, since the timer already exists with tags named tag1 and tag2.

In your use case what are the differing tag keys?

This use case used to throw an exception, but now I believe it merely logs a warning.

[shakuzen]

This is expected behavior currently due to the limitation that the Prometheus client does not allow meters with the same name and a different set of labels. There is #877 to see if we can work around the limitation in the Prometheus client. Otherwise we suggest you abide by the limitations set by the Prometheus client. It's unfortunate portability-wise since other registries tend to not have this limitation.
Duplicate of #877

[mlubavin-vg]

Thank you @checketts and @shakuzen . I've subscribed to #877

[angry-cellophane]

It's not a limitation of the prometheus client but just the way the prometheus-micrometer integration is written.
It is totally possible to collect metrics with different tags, MicrometerCollector needs to be changed to store label names dynamically like it does for label values. I raised a PR with a basic fix to the issue #2653