Alert rule is showing that the expression is satisfied. However Alert is not firing

[KR411-prog]

Alert rule is showing that the expression is satisfied. However Alert is not firing.

Here is the alert rule:
apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: natgw-alert-rules namespace: {{ .Values.namespace }} labels: prometheus: k8s role: alert-rules spec: groups: - name: natgw-alert-rules rules: - alert: NatGWReservedFIPFailures expr: | increase( nat_gw_errors_total{error_type="nat_reserved_fip_failed"}[5m] ) > 0 #for: 1m labels: severity: medium annotations: summary: "NAT GW reserved FIP failure" description: "NAT GW reserved FIP failures are occurring in the last 5 minutes"

[TheMeier]

Maybe the alert is silenced, inhibited or muted? Which web-ui is that?

[KR411-prog]

Its not muted or inhibited or silenced. Its Openshift console.

[TheMeier]

Hm hard to tell like this. Can you dump the alerts from the api endpoint and check if you see something there?
/api/v2/alerts?silenced=true&inhibited=true&muted=true&active=true
Also you could check via promql if prometheus is firing ALERTS{alertname="NatGWReservedFIPFailures", alertstate="firing"}

If it's firing in prometheus but not aletmanager, maybe the prometheus can't talk to alertmanager

[TheMeier]

well that looks like the alert is not fireing in prometheus then.

[KR411-prog]

yes..but the metrics is showing correctly. This alert rule expression also satisfying the count and expression

[KR411-prog]

I have placed servicemonitor in the app namespace and prometheusrules in userworkload-monitoring namespace.
All the other alerts are firing and working correctly.

One thing I suspect is ,
if I replace nat_gw_errors_total with some kubernetes prometheus metrics such as kube_pod_container_status_restarts_total, it works.
So the issue could be with that metric nat_gw_errors_total itself.

I have exposed this metrics from my controller code as below,

var (
	natGWResourcesCreated = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "nat_gw_resources_created_total",
			Help: "Total NAT GW resources created",
		},
		[]string{"namespace", "cluster"},
	)

	natGWErrors = prometheus.NewCounterVec(
		prometheus.CounterOpts{
			Name: "nat_gw_errors_total",
			Help: "Total NAT GW errors",
		},
		[]string{"error_type", "namespace"},
	)
)


if reservefiperr != nil {
			//It prints the error but continues further to create port, a different available fip and associate fip with it
			natGWErrors.WithLabelValues("nat_reserved_fip_failed", namespace).Inc()

			l.Error(reservefiperr, "Failed to use reserved floating IP")
		}

By exposing this metrics in 8080 port of deployment object, its even showing the metrics in /metrics

I am not sure what else I am missing

[awsxchange-pixel]

Thank you

[awsxchange-pixel]

Thank you

[TheMeier]

The graph you posted in #4904 (reply in thread) shows a constant value but not an increase over time. Also you might need a keep_firing_for clause, otherwise your alert will only be active for 5 minutes max in the case there is actually an increase.

[KR411-prog]

ok..But my expression is only,
sum(nat_gw_errors_total{error_type="nat_reserved_fip_failed"}) by (namespace) > 0

And it shows in the graph as,

which has satisfied the rule.

I didnt add any "increase" in the expression

[TheMeier]

Here is the alert rule:
apiVersion: monitoring.coreos.com/v1 kind: PrometheusRule metadata: name: natgw-alert-rules namespace: {{ .Values.namespace }} labels: prometheus: k8s role: alert-rules spec: groups: - name: natgw-alert-rules rules: - alert: NatGWReservedFIPFailures expr: | increase( nat_gw_errors_total{error_type="nat_reserved_fip_failed"}[5m] ) > 0 #for: 1m labels: severity: medium annotations: summary: "NAT GW reserved FIP failure" description: "NAT GW reserved FIP failures are occurring in the last 5 minutes"

^^ there is an increase()

[KR411-prog]

I changed that already..My latest alert rule is as given below. I just updated expression as increase didnt help. So I tried sum.

- alert: NatGWReservedFIPFailures2
          expr: sum(nat_gw_errors_total{error_type="nat_reserved_fip_failed"}) by (namespace) > 0 
          for: 0m
          labels:
            severity: medium
            cluster: {{ .Values.cluster_name }}
          annotations:
            message: Kubernetes NAT GW reserved FIP failures

For this, the result is,
sum(nat_gw_errors_total{error_type="nat_reserved_fip_failed"}) by (namespace) > 0

[TheMeier]

[KR411-prog]

Finally it worked. After keeping both servicemonitor and prometheusrule in the same app namespace.