hey @joshleecreates, I was still not able to see the RED metrics in Grafana.
All charts are with No Data.

hey @joshleecreates, I was still not able to see the RED metrics in Grafana. All charts are with No Data.

It worked once for me but I am now seeing the same. I suspect that when it worked I was working off of the branch with the span filtering rules to reduce cardinality, I'll test again after that is merged.

I'm still seeing issues with Prometheus with the cardinality fix merged:

prometheus  | ts=2024-07-16T15:58:30.576Z caller=manager.go:163 level=info component="rule manager" msg="Starting rule manager..."

prometheus  | ts=2024-07-16T16:21:51.834Z caller=write_handler.go:134 level=error component=web msg="Out of order sample from remote write" err="out of order sample" series="{__name__=\"target_info\", container_id=\"3e0bddb9d06cf94fb84a2d772930cc64db46193213180c29951a1ee887391103\", docker_cli_cobra_command_path=\"docker compose\", host_arch=\"aarch64\", host_name=\"3e0bddb9d06c\", job=\"quoteservice\", os_description=\"6.6.32-linuxkit\", os_name=\"Linux\", os_type=\"linux\", os_version=\"#1 SMP Thu Jun 13 14:13:01 UTC 2024\", process_command=\"public/index.php\", process_command_args=\"[\\\"public/index.php\\\"]\", process_executable_path=\"/usr/local/bin/php\", process_owner=\"www-data\", process_pid=\"7\", process_runtime_name=\"cli\", process_runtime_version=\"8.3.9\", service_version=\"1.0.0+no-version-set\", telemetry_distro_name=\"opentelemetry-php-instrumentation\", telemetry_distro_version=\"1.0.3\", telemetry_sdk_language=\"php\", telemetry_sdk_name=\"opentelemetry\", telemetry_sdk_version=\"1.0.8\"}" timestamp=1721146894925

There's a 23 minute gap before the error appears. I checked in the beginning of that time frame and didn't see any span metrics (or Grafana errors).

I think the changes in this PR are necessary for Grafana but there is still something else going on with Prometheus.

That second issue is different from Prometheus, and it existed before. #1622 is the related issue for it.

I pushed a couple of fixes to your branch for that Prometheus issue, and to use a 2m interval instead of 1m. I guess with a 1m metric, we need at least 2 samples for rate to work.

I think that Prometheus change fixed out of order samples, but now we are getting out of order exemplars.

prometheus  | ts=2024-07-19T13:05:38.103Z caller=write_handler.go:175 level=warn component=web msg="Error on ingesting out-of-order exemplars" num_dropped=1

Some searching tells me this particular thing is not yet solved in Prometheus, and there is an open issue for it.

I've had this branch running for 4+ days, and Prometheus is stable, with the dashboards working as intended.

We still have an error in Prometheus for out of order exemplars, but that is a known issue with Prometheus, that we should track separately.

@julianocosta89 can you take another look to see if this works for you?

🥳 thanks @joshleecreates and @puckpuck!
It seems we have charts working again

Thanks @puckpuck for wrapping this up!