loki.process forwarding to multiple components blocks all destinations if one is blocked

[claytonpeters]

What's wrong?

We have a loki.process component in our setup which for testing we had set to forward_to multiple endpoints as we needed K8s logs to go to multiple places. When one of the loki.write components in this block was unable to send on metrics (because the server it was sending to was temporarily unavailable), we noted that the other components that loki.process was forwarding on to also stopped receiving logs.

Steps to reproduce

• Setup alloy to take a Loki source (for example from Kubernetes pod logs, i.e. loki.source.kubernetes)
• Have a loki.process component take those logs and forward them on to multiple write end points (a loki.write, and at least one other)
• Take down the destination server for one of those write loki.write end points so that the logs fail to deliver
• All destinations will now stop receiving logs

System information

Linux 5.15.0-122

Software version

Grafana Alloy v1.4.2

Configuration

discovery.kubernetes "pods" {
  role = "pod"
}
discovery.relabel "pods_logs" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    action        = "replace"
    target_label  = "namespace"
  }
  // ..etc..
}
loki.source.kubernetes "pods_logs" {
  targets    = discovery.relabel.pods_logs.output
  forward_to = [loki.process.pods_logs.receiver]
  clustering {
    enabled = true
  }
}
loki.process "pods_logs" {
  stage.static_labels {
    values = {
      cluster = "cluster-name",
    }
  }
  forward_to = [
    loki.write.default.receiver,
    loki.write.second_test.receiver,
  ]
}
loki.write "default" {
  endpoint {
    url       = "http://default-loki-endpoint:3100/loki/api/v1/push"
    tenant_id = "public"
  }
}
loki.write "second_test" {
  endpoint {
    url       = "http://second-loki-endpoint:3100/loki/api/v1/push"
    tenant_id = "public"
  }
}

Logs