Should the SnippetsFilter status contain the controller name?

[kate-osborn]

Many NGF controllers may share a SnippetsFilter resource.

For example, say you are upgrading NGF to a new version and running both simultaneously. The controllers will have different controller names, say:

• gateway.nginx.org/nginx-gateway-controller-1.3
• gateway.nginx.org/nginx-gateway-controller-1.4

Now, let's say you have the following Route:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: coffee
spec:
  parentRefs:
  - name: gateway # owned by gateway.nginx.org/nginx-gateway-controller-1.3
    sectionName: http
  hostnames:
  - "cafe.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    filters:
    - type: ExtensionRef
      extensionRef:
        group: gateway.nginx.org/v1alpha1
        kind: SnippetsFilter
        name: access-control
    backendRefs:
    - name: headers
      port: 80

which references the following SnippetsFilter:

apiVersion: gateway.nginx.org/v1alpha1
kind: SnippetsFilter
metadata:
  name: access-control
spec:
  snippets:
  - context: http.server.location
    value: |
      allow 10.0.0.0/8;
      deny all;

The Route has the following status:

Status:
  Parents:
    Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               The route is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               All references are resolved
      Observed Generation:   1
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         gateway.nginx.org/nginx-gateway-controller-1.3
    Parent Ref:
      Group:         gateway.networking.k8s.io
      Kind:          Gateway
      Name:          gateway
      Namespace:     default
      Section Name:  http

and the SnippetsFilter:

Status:
  Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               SnippetsFilter is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted

Now, you want to attach this Route to the Gateway that is owned by the new version of NGF, so you update the Route and add the new Gateway:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: coffee
spec:
  parentRefs:
  - name: gateway # owned by gateway.nginx.org/nginx-gateway-controller-1.3
    sectionName: http
  - name: gateway-1.4 # owned by gateway.nginx.org/nginx-gateway-controller-1.4
    sectionName: http
  hostnames:
  - "cafe.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /coffee
    filters:
    - type: ExtensionRef
      extensionRef:
        group: gateway.nginx.org/v1alpha1
        kind: SnippetsFilter
        name: access-control
    backendRefs:
    - name: headers
      port: 80

the SnippetsFilter doesn't change.

Now, the status of the Route is updated and shows the status for both controllers:

Status:
  Parents:
    Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               The route is accepted
      Observed Generation:   2
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               All references are resolved
      Observed Generation:   2
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         gateway.nginx.org/nginx-gateway-controller-1.3
    Parent Ref:
      Group:         gateway.networking.k8s.io
      Kind:          Gateway
      Name:          gateway
      Namespace:     default
      Section Name:  http
    Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               The route is accepted
      Observed Generation:   2
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               All references are resolved
      Observed Generation:   2
      Reason:                ResolvedRefs
      Status:                True
      Type:                  ResolvedRefs
    Controller Name:         gateway.nginx.org/nginx-gateway-controller-1.4
    Parent Ref:
      Group:         gateway.networking.k8s.io
      Kind:          Gateway
      Name:          gateway
      Namespace:     default
      Section Name:  http

so you can tell the Route was processed by the gateway.nginx.org/nginx-gateway-controller-1.4 controller and is accepted.

However, the SnippetsFilter is just a list of conditions with no controller name. This list acts as a map, where the key is the condition type. Therefore, only one condition for the Accepted condition type can exist. So even though both NGF controllers will update the SnippetsFilter status, only one of their conditions will "win":

Status:
  Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               SnippetsFilter is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted

We don't know which controller wrote this status, and it's possible that the gateway.nginx.org/nginx-gateway-controller-1.4 controller did not accept the SnippetsFilter and its status update was overwritten by the gateway.nginx.org/nginx-gateway-controller-1.3 controller.

Proposed Solution

Add controller name to the status:

type SnippetsFilterStatus struct {
    Controllers []ControllerStatus
}

type ControllerStatus struct {
    ControllerName string
    Conditions []metav1.Conditions
}

this would change the status of the SnippetsFilter in the above example to:

Status:
  Controller Name: gateway.nginx.org/nginx-gateway-controller-1.3
  Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               SnippetsFilter is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted
  Controller Name: gateway.nginx.org/nginx-gateway-controller-1.4
  Conditions:
      Last Transition Time:  2024-09-11T16:08:26Z
      Message:               SnippetsFilter is accepted
      Observed Generation:   1
      Reason:                Accepted
      Status:                True
      Type:                  Accepted

[mpstefan]

I can definitely see this happening in clusters with more than one instance of NGF. Should be less of a problem once we support multiple gateways, as I do believe that will be simply the best way to set up those multi-tenanted clusters.

We should make an issue for this, and depending on the effort involved, prioritize the change.

[kate-osborn]

Converted to issue