Tracking: Address Clustering Issues

[thampiotr]

Request

There are a few issues that users report and we're observing, that can lead to data issues: 1) there can be gaps in metrics under some circumstances when instances join cluster, 2) there can be elevated errors and alerts when writing to TSDB in some cases, 3) there can be duplicated metrics in other cases.

The extent of these issues is not high, but since these are potential data loss issues, we want to address them and fully understand the problem.

Use case

Sending data should not be dropped.

Tasks

Preview Give feedback

1. clustering: deal with gaps during cluster changes #249
   bug enhancement frozen-due-to-age +3
   
2. [Clustering] Filter the label "instance" from the hash computation agent#6792
   bug frozen-due-to-age needs-attention variant/flow +4
3. Review in detail the errors when remote writing and identify the most common ones
4. remote_write: add histogram info to error logs in queue_manager prometheus/prometheus#14326
   +0
5. Add SR on dashboards mixin: Add SR to remote_write dashboard #1100
6. Verify dashboards latest changes
7. Create an issue to track the known OOO problems.
8. Review reports of duplicate samples errors
9. Investigate resizing causing out-of-order errors
10. Review reports of gaps in metrics
11. Reduce spammy logging relatd to clustering: https://github.com/grafana/alloy/blob/881c4b7cf72d0a9c068cca04225fac042e6e4714/internal/service/cluster/cluster.go#L220
12. Rate limit how often components get notified about cluster changes #1261
    frozen-due-to-age +1
13. [clustering] failure to discover peers when the number of instances and exposed ports is large #1208
    7 of 7
    frozen-due-to-age pir-action-item +2
    
14. [stretch] explore options for clustering support for push-based workflows
15. [stretch] New highly-available cluster architecture POC
16. [follow-up] make cluster improvements GA: Graduate the improvements to clustering to GA once proven #1274
17. [follow-up] verify we address this: alloy metrics pod crashes with alloy v1.2.1 when using prometheus.operator.servicemonitors config #1349
18. [new/stretch] don't admit traffic until cluster is sufficient size: Proposal: Only attempt to send metrics after joining a sufficiently large cluster #201
19. [stretch] Fine-grained component scheduling: clustering: allow cluster-aware component scheduling #399
20. [stretch] When using clustering, exporters may not work correctly due to instance label #1009

[diguardiag]

I am noticing this behaviour on a k8s cluster (~1800 pods), with an alloy cluster of 3, Istio present, and pod autodiscovery enabled.

• Pods have 13GB Memory each (upper limit)
• Peers change constantly (but they are always up and running) (screen below)

We're experiencing:

• metrics data loss
• a lot of lines with Dropped sample for series that was not explicitly dropped via relabelling
• cluster pods not agreeing on who is part of the cluster (see screen)
  

[gowtham-sundara]

I noticed an issue today where one of the pods fell out of the clustering, it's present in discovery but none of the alloy pods actually scrape them. This didn't go away over a long period of time so I am not sure if it's related to #1 mentioned.

[christopher-wong]

This is my Helm configuration for my deployment of Alloy (24 nodes, so 24 Alloy pods). When I enable clustering, alloy.clustering.enabled = true, metrics stop being scraped altogether.

alloy:
  configMap:
    content: |-
      prometheus.remote_write "default" {
        endpoint {
          url = "http://mimir-gateway.monitoring.svc:80/api/v1/push"
        }
      }

      prometheus.operator.servicemonitors "services" {
        forward_to = [prometheus.remote_write.default.receiver]

        clustering {
          enabled = true
        }
      }

      prometheus.operator.podmonitors "pods" {
        forward_to = [prometheus.remote_write.default.receiver]

        clustering {
          enabled = true
        }
      }
  clustering:
    enabled: false      
  resources:
    requests:
      cpu: 100m
      memory: 2Gi
    limits:
      cpu: 1.5
      memory: 12Gi
configReloader:
  resources:
    requests:
      cpu: "1m"
      memory: "5Mi"
    limits:
      cpu: 10m
      memory: 10Mi

[itjobs-levi]

I have Alloy Agent replica 3 ea (CPU: 1000m / Memory: 4Gi)
Enable clustering mode (both scrape exporters)
Configure to scrape the unix exporter and process exporter on about 200 servers at one-minute intervals.
When scraping, many errors such as err-mimir-duplicate-label-names occur in mimir.
In the Grafana mimir document, err-mimir-duplicate-label-names appears to be a problem caused by existing records.
I think this is caused by cluster splitting job load balancing.

The first is that this feels like a load, is that correct?
If it's not a secondary load, is it possible to turn off these logs?

[thampiotr]

@itjobs-levi @christopher-wong @gowtham-sundara @diguardiag could you open issues for these and provide clear steps to reproduce? These may need to be looked into separately.

[thampiotr]

Closing this as done. The stretch goals are left as issues in their own rights.