Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Distribution-to-Histogram conversion yields invalid data #25

Closed
cboggs opened this issue Apr 12, 2018 · 2 comments
Closed

Distribution-to-Histogram conversion yields invalid data #25

cboggs opened this issue Apr 12, 2018 · 2 comments

Comments

@cboggs
Copy link

cboggs commented Apr 12, 2018 •
edited
Loading

Setup

Run stackdriver_exporter with --monitoring.metrics-type-prefixes=bigtable.googleapis.com.

Expected

Metrics like stackdriver_bigtable_table_bigtable_googleapis_com_server_latencies_count should present as monotonically increasing counters, as would normal Prometheus histogram counters.

Actual

Metrics stackdriver_bigtable_table_bigtable_googleapis_com_server_latencies_count frequently decrement and oscillate in ways they shouldn't.

Screenshot:
screen shot 2018-04-12 at 9 10 57 am

This inherently throws off quantile calculations on the respective histogram buckets, as the individual bucket counts oscillate in the same manner. This yields per-bucket rates that provide errant quantiles (ex: 47-day-long latency measurements on p50).
@cboggs cboggs changed the title Duration-to-Histogram conversion yields invalid data Distribution-to-Histogram conversion yields invalid data Apr 12, 2018
@nilebox
Copy link

nilebox commented Jul 29, 2021

It's not just _count, but bucket values as well.
The problem is due to DISTRIBUTION + DELTA metrics in Cloud Monitoring (Stackdriver) returning deltas for a given interval, whereas Prometheus counter should present the total accumulated count.

Related: #116 (comment)

@osabina osabina mentioned this issue Jul 11, 2022
Closed
@kgeckhart kgeckhart mentioned this issue Aug 26, 2022
Merged
SuperQ pushed a commit that referenced this issue Nov 22, 2022
@kgeckhart
Track delta metrics between scrapes (#168)
42321da 
This PR introduces support for aggregating DELTA metrics in-memory. This implementation is somewhat based on #167 but with extra functionality which is intended to solve the problem identified here, #167 (comment).

I chose to inject interfaces for the delta and distribution stores at the Collector level to try to keep library friendliness intact, #157

I did as much as I could to explain the solution and the tradeoffs with aggregating deltas in this manner in the README.md. The bulk of the changes are in `monitoring_metrics.go` where I tried to obfuscate as much of the delta logic as possible.

Should hopefully resolve #116 and #25 

Signed-off-by: Kyle Eckhart <[email protected]>
@kgeckhart
Copy link
Contributor

kgeckhart commented May 26, 2023 •
edited
Loading

@SuperQ I think this one can be closed. I think I got syntax wrong on #168 for it to auto close when it merged

@SuperQ SuperQ closed this as completed May 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@SuperQ @cboggs @kgeckhart @nilebox