watchNamespaceSelectors does not create dynamic dashboards in grafana.

[chris530]

Gloo Edge Product

Enterprise

Gloo Edge Version

v1.18.2

Kubernetes Version

v1.31

Describe the bug

Customer created a ticket ( https://solo-io.zendesk.com/agent/tickets/5239 ). Upstream dashboards in grafana do not get created when using watchNamespaceSelectors in Settings. I was able to reproduce this issue in my environment as well.

``

Expected Behavior

Defining watchNamespaceSelectors should find namespaces and add dynamic dashboards.

Steps to reproduce the bug

Installed gloo ( 1.18.2 ) , where no watchNamespaceSelectors is defined in Settings.

Added a static upstream , which is accepted from Gloo.

---
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
  name: nginx-static-80
  namespace: chris
spec:
  connectionConfig:
    maxRequestsPerConnection: 0
    perConnectionBufferLimitBytes: 777666
  static:
    hosts:
      - addr: nginx.web.svc.cluster.local
        port: 80

Logging into grafana, a upstream dashboard was created.

I reinstall gloo with same steps. Then label namespace.

kubectl create ns chris 
kubectl label ns chris gloo-translate=enabled

Confirmed label selector is working

kubectl get ns -l gloo-translate=enabled                     
NAME    STATUS   AGE
chris   Active   3m

Added label selector to settings.

...
  refreshRate: 60s
  watchNamespaceSelectors:
  - matchLabels:
      gloo-translate: enabled

Waiting awhile, ( also restarted observability pod ). upstream dashboard never gets created.

NOTE: If I remove the watchNamespaceSelectors in Settings , the dashboard gets immediately created in grafana.

Additional Environment Detail

Gloo 1.18.2
K8s 1.31
EKS

Additional Context

Followed these steps

https://docs.solo.io/gloo-edge/main/operations/upgrading/faq/
https://docs.solo.io/gloo-edge/main/installation/advanced_configuration/multiple-gloo-installs/#specify-namespaces-to-watch-for-kuberenetes-services-and-gloo-gateway-crs

[chris530]

Also tried with a expression , with no luck.

  watchNamespaceSelectors:
  - matchLabels:
      gloo-translate: enabled
  - matchExpressions:
    - key: env
      operator: In
      values:
      - prod

kubectl get ns -l gloo-translate=enabled -l env=prod
NAME    STATUS   AGE
chris   Active   6h26m

kubectl get upstream -n chris
NAME              AGE
nginx-static-80   6h9m