[Bug] unabel to create notification policy

[loveisall1995]

Describe the bug
When applying a GrafanaNotificationPolicy, the operator logs indicate that it skips reconciliation due to an existing notification policy being applied to the same instance. This prevents new policies from being configured as expected.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaNotificationPolicy
metadata:
  name: engg-infra-policy
spec:
  instanceSelector:
    matchLabels:
      instance: grafana
  route:
    receiver: uat-bob-engg-infra-alerts
    group_by:
      - grafana_folder
      - alertname
    routes:
      - receiver: uat-bob-engg-infra-alerts
          - - team
            - =
            - engg-infra 

Helm Version
operator version: 5.12.0
grafana version: v5.15.1

grafana operator logs:

2025-01-02T12:56:39Z	INFO	GrafanaNotificationPolicyReconciler	instance already has a different notification policy applied - skipping	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy", "GrafanaNotificationPolicy": {"name":"uat-bob-engg-infra-policy","namespace":"grafana"}, "namespace": "grafana", "name": "uat-bob-engg-infra-policy", "reconcileID": "4ef43454-7f48-4deb-b8b5-ac9ca067b9e8", "grafana": "grafana"}

[theSuess]

Notification policies are global objects, so you cannot apply more than one. Is the operator correct in that you have another notification policy configured as well?

We have an open proposal for dynamic notification policies in #1800, but this is not yet implemented.

[loveisall1995]

@theSuess I have created only one notification policy using the Grafana Operator. However, there is a default notification policy already present.

[theSuess]

Hm, this is weird then. The default notification policy should not interfere, as the detection is based off the operator.grafana.com/applied-notificationpolicy label (set on the Grafana resource). Can you check the contents of that? It should point to the notification policy the operator thinks is currently applied.

[loveisall1995]

@theSuess, I've updated the notification policy based on the documentation from [Grafana Operator Alerting.](https://grafana.github.io/grafana-operator/docs/alerting/) I’ve also ensured the label is set correctly. Below is the current configuration but again facing the same issue. All other resources are working except for the notification policy.

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaNotificationPolicy
metadata:
  name: notification-policies
spec:
  instanceSelector:
    matchLabels:
      instance: "grafana"
  route:
    receiver: grafana-default-email
    group_by:
      - grafana_folder
      - alertname
    routes:
      - receiver: operations-team
        object_matchers:
          - - team
            - =
            - operations
        routes:
          - object_matchers:
              - - severity
                - =
                - high
            repeat_interval: 5m
      - receiver: security-team
        object_matchers:
          - - team
            - =
            - security

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    instance: grafana
spec:
  external:
    url: http://grafana.grafana.svc.cluster.local
    adminPassword:
      name: grafana-secrets
      key: GRAFANA_ADMIN_PASSWORD
    adminUser:
      name: grafana-secrets
      key: GRAFANA_ADMIN_USER

grafana operator logs:

2025-01-07T10:16:00Z	INFO	GrafanaNotificationPolicyReconciler	instance already has a different notification policy applied - skipping	{"controller": "grafananotificationpolicy", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaNotificationPolicy", "GrafanaNotificationPolicy": {"name":"notification-policies","namespace":"grafana"}, "namespace": "grafana", "name": "notification-policies", "reconcileID": "edce34e1-25b6-4669-9a54-ca1490a3b78f", "grafana": "grafana"}

[theSuess]

Can you see what the value of the operator.grafana.com/applied-notificationpolicy label is after applying these resources?

[loveisall1995]

@theSuess, The label remained the same even after applying, so I tried recreating Grafana. The notification policy has now been created. Thanks for your support!

apiVersion: grafana.integreatly.org/v1beta1
kind: Grafana
metadata:
  name: grafana
  labels:
    instance: grafana
spec:
  external:
    url: http://grafana.grafana.svc.cluster.local
    adminPassword:
      name: grafana-secrets
      key: GRAFANA_ADMIN_PASSWORD
    adminUser:
      name: grafana-secrets
      key: GRAFANA_ADMIN_USER