[WIP] A prototype implementation to improve startup performance

[M00nF1sh]

Issue

#3326

Description

DON'T MERGE THIS
Built image for test: m00nf1sh/aws-load-balancer-controller:v2.8.3-checkpoint

Intro

This is a prototype implementation to improve performance during controller startup. The root cause is during restart(e.g. leadership change or hardware issues on node), the controller will reconcile all existing Ingress and Services in cluster. In some large clusters with a lot Ingress/Services, this could take a long time due to AWS API throttles, thus impacting the ability to handle other events(e.g. pod deployments).

Design

The idea is to save the "last reconciled state" as annotations into Ingress and Service objects, (potentially TargetGroupBindings) as well. So during controller restart, it can compare the current state vs "last reconciled state"(from annotation), and skip reconcile on already reconciled resources.
With this implementation, the "last reconciled state" is computed as sha256 of the ELBv2 JSON model built for Ingress and Services. For TargetGroupBindings, it can be the list of current backend targets(TODO, need some refactor).

Alternative design considered:

1. alternatively, the "last reconciled state" can be the Ingress & Service's annotation & spec itself. I originally tried this approach but later realized that Ingress's configuration(ALB) can be impacted by Service and even Secrets as well(e.g. OIDC). So if we only record Ingress itself's annotation & spec as "last reconciled state", then we won't be able to trigger updates when Service/Secrets were changed(e.g. during the gap when controller restart)

Next Steps

1. refactor the Ingress&Service controller's code to make the code more cleaner.
2. The Marshaled JSON don't contain OIDC clientID/secret, so to trigger Ingress updates for OIDC secret changes, we need to include them in the digest somehow.
3. add checkpoint support for targetGroupBinding
4. [best to have] inject a custom work queue implementation into controllers where we can still enqueue those unchanged Ingress & Service during restart, but reconcile them at a lower priority and limited enqueue rate limit.

Checklist

• Added tests that cover your change (if possible)
• Added/modified documentation as required (such as the README.md, or the docs directory)
• Manually tested
• Made sure the title of the PR is a good description that can go into the release notes

BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯

• Backfilled missing tests for code in same general area 🎉
• Refactored something and made the world a better place 🌟

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: M00nF1sh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [M00nF1sh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[k8s-ci-robot]

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.