Gateway and HTTPRoute unable to be deleted -- stuck finalizing

[ellistarn]

Reproduction steps:

1. Create gateway and gateway class
2. Delete gateway class
3. Delete gateway (stuck deleting)
4. Delete HTTPRoute (stuck deleting)
5. Observe finalizer still on resource.

This log line looks like the culprit.

gateway-api-controller-6d499b898c-nq4sr manager 2023-03-05T01:56:33.540Z	INFO	controller.gateway	Ignore it since not link to any gatewayclass	{"reconciler group": "gateway.networking.k8s.io", "reconciler kind": "Gateway", "name": "carverspring-2-v1bayfdckt", "namespace": "default"}

- apiVersion: gateway.networking.k8s.io/v1alpha2
  kind: HTTPRoute
  metadata:
    creationTimestamp: "2023-03-03T23:47:55Z"
    deletionGracePeriodSeconds: 0
    deletionTimestamp: "2023-03-04T02:28:55Z"
    finalizers:
    - httproute.k8s.aws/resources

[aaroniscode]

This may be due to #105 . I'm working on the PR now to add the finalizer for GatewayClass

[liwenwu-amazon]

Today, we expect amazonp-vpc-lattice GatewayClass is an object managed by Infrastructure provider and kind of like the capability of a EKS cluster. We are not expecting a cluster operator to be able to delete it.

Especially, we want to prevent it from getting deleted (#105) and all the VPC lattice resource get accidently wiped outs.

[aaroniscode]

Hi @liwenwu-amazon -- I'm not sure I understand how we would prevent a resource from being deleted. And as I mentioned in #105 , the community is likely not moving forward with finalizers for GatewayClass so we would want a mechanism to prevent system hang in case someone deletes the GatewayClass.

One alternative is to have the controller re-create the GatewayClass if deleted?

[liwenwu-amazon]

Not sure what the semantic meaning of deleting amazon-vpc-lattice gatewayclass? Does it mean the EKS cluster no longer support VPC lattice?

[liwenwu-amazon]

I am hoping in long-run, the lattice controller will be an automatic EKS-cluster add-on. And EKS-cluster always support AWS VPC lattice. AWS EKS infrastructure automatically create amazon-vpc-lattice gateway class

[ellistarn]

The kubernetes pattern is typically to allow arbitrary resource application and deletion, and handle these issues at runtime. It's common for use cases to install or uninstall yaml in any order, and validating the existence of interdependent resources will break many flows (e.g. helm, flux). This is in strong contrast to AWS APIs, which are highly interdependent.

As linked by @aaroniscode, the upstream spec https://gateway-api.sigs.k8s.io/references/spec/#gateway.networking.k8s.io%2fv1beta1.GatewayClass states that the gateway implementation must put a finalizer on the gateway, to govern deletion of the gateway class. It's not clear to me how this is supposed to work in practice. e.g., what happens if the user uninstalls the gateway implementation? Who is responsible for removing the finalizer?

Maybe we can reach out to sig-networking to get an authoritative answer from the spec designers.

edit: just realized @aaroniscode already did this here: #105 (comment)

Perhaps we should bundle the gateway class as part of the helm install, and never require a user to create one at runtime.

[ellistarn]

Discussed w/ @liwenwu-amazon. As a path forward, we will include the gateway class as part of the installation. Users will not need to create this manually. @aaroniscode, thoughts?

[aaroniscode]

Sounds good to me @ellistarn . I think other projects have taken this approach as well. The question remains what to do if the GatewayClass is deleted. I mentioned here that the community is likely not going forward with a finalizer and no other projects appear to be implementing one.

If that's the case, to avoid a hung system with the GatewayClass deleted, I see two paths:

• Controller reconciles the GatewayClass and puts it back if deleted
• Controller should consider HTTPRoute and Gateway in scope if they have the VPC Lattice finalizer on them and they are deleted.

What do you think?

[hu1buerger]

Is a workaround known to remove the GatewayClass?