*: Bump k8s.io and controller-runtime dependencies #10069

timflannagan · 2024-09-19T19:03:02Z

Description

This PR bumps several critical Go direct dependencies including k8s.io, go-control-plane, and controller-runtime.

Code changes

Reverts solo-io#7920 in the kube-based upstream plugin. Previously, we were typecasting our own lister abstraction that sits on top of client-go into concrete corev1 listers. That approach was breaking our regression suite -- likely because upstream refactored lister generation to be generic in 1.31.x -- and required us to revert that PR in order to get the regression suite back online. From my perspective, this type of typecasting violates our own abstraction layer and any performance-related issues should've been tackled closer to the source-of-truth (e.g. solo-kit).

Additionally, updates to pkg/bootstrap/leaderelector/kube/metrics.go were required after bumping controller-runtime to 0.18.x / 0.19.x to get builds back online.

CI changes

Update our k8s N-3 matrix from 1.29 to 1.31. Similarly, update our minimum k8s version from 1.25 to 1.27.
Update any unit tests that were comparing proto structs via To(Equal(...) type operations. This is needed due to bumping the protobuf dependency as the default, underlying sizeCache value was changed. Further investigation into why this is needed will be tackled as a follow-up

Context

This is blocking several streams of downstream work including RFE and GW API integration initiatives.

Notes for reviewers

Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added tests that prove my fix is effective or that my feature works

BOT NOTES:
Related to solo-io#9683

Signed-off-by: timflannagan <[email protected]> Co-authored-by: Sam Heilbron <[email protected]> Co-authored-by: Tyler Schade <[email protected]>

Signed-off-by: timflannagan <[email protected]>

solo-changelog-bot · 2024-09-19T19:44:41Z

Issues linked to changelog:
solo-io#9683

Signed-off-by: timflannagan <[email protected]>

Note: these clients were manually generated using a solo-kit that points to my local fork that implements kgateway-dev#564. The gateway & gloo clients were updated to adopt recently support for generics throughout the 1.31 client-go release. Namely, listers and clients adopt this new approach. The nested extauth and graphql APIs have updated hack/update-codegen.sh bash scripts checked in with this commit, but I think we need to update the solo-kit.json configuration for those directories since we weren't previously committing their k8s clients. Similarly, the "gloosnapshot" API doesn't need k8s clients generated too. Signed-off-by: timflannagan <[email protected]>

Signed-off-by: timflannagan <[email protected]>

timflannagan · 2024-09-20T16:20:57Z

.github/workflows/.env/nightly-tests/max_versions.env

-node_version='v1.29.2@sha256:51a1434a5397193442f0be2a297b488b6c919ce8a3931be0ce822606ea5ca245'
-kubectl_version='v1.29.2'
-kind_version='v0.20.0'
+node_version='v1.31.0@sha256:53df588e04085fd41ae12de0c3fe4c72f7013bba32a20e7325357a1ac94ba865'


Quick note: go.mod specifies the 1.31.1 patch version, but I didn't see a 1.31.1 sha image in the kind releases, so I left this here. I don't think it matters too much w.rt. patch version skew between k8s server and client versions.

timflannagan · 2024-09-20T16:21:40Z

.github/workflows/.env/nightly-tests/min_versions.env

@@ -1,6 +1,6 @@
-node_version='v1.25.16@sha256:5da57dfc290ac3599e775e63b8b6c49c0c85d3fec771cd7d55b45fae14b38d3b'
-kubectl_version='v1.25.16'
+node_version='v1.27.3@sha256:3966ac761ae0136263ffdb6cfd4db23ef8a83cba8a463690e98317add2c9ba72'


We're jumping from 1.29 to 1.31 in Gloo, so updating the min supported k8s version to 1.27 to maintain the N-3 matrix.

timflannagan · 2024-09-20T16:21:53Z

changelog/v1.18.0-beta23/9683-k8s-1.31.yaml

@@ -0,0 +1,12 @@
+changelog:


TODO: Fix this changelog.

timflannagan · 2024-09-20T16:22:25Z

pkg/bootstrap/leaderelector/kube/metrics.go

@@ -35,3 +35,7 @@ func (s *switchAdapter) On(name string) {
 func (s *switchAdapter) Off(name string) {
 	s.gauge.WithLabelValues(name).Set(0.0)
 }
+
+func (s *switchAdapter) SlowpathExercised(name string) {


Needed for controller-runtime 0.18.x due to client-go leaderelection changes.

timflannagan · 2024-09-20T16:23:40Z

projects/gateway/pkg/api/v1/kube/hack/update-codegen.sh

@@ -10,10 +10,9 @@ ROOT_PKG=github.com/solo-io/gloo/projects/gateway/pkg/api/v1
 CLIENT_PKG=${ROOT_PKG}/kube/client
 APIS_PKG=${ROOT_PKG}/kube/apis

-# Below code is copied from https://github.com/weaveworks/flagger/blob/master/hack/update-codegen.sh


This file is generated by solo-kit. We updated the Go template file in solo-io/solo-kit#560. Note, this file is technically bugged and I have an open PR for fixing this in solo-io/solo-kit#564.

timflannagan · 2024-09-20T16:24:09Z

projects/gateway/pkg/validation/robust_client_test.go

@@ -65,7 +66,7 @@ var _ = Describe("RetryOnUnavailableClientConstructor", func() {
 		// sanity check
 		resp, err := client.Validate(rootCtx, &validation.GlooValidationServiceRequest{})
 		Expect(err).NotTo(HaveOccurred())
-		Expect(resp).To(Equal(res))
+		Expect(resp).To(matchers.MatchProto(res))


Needed due to the protobuf jump that was clumped with these dependency bumps.

timflannagan · 2024-09-20T16:24:39Z

projects/gateway2/controller/controller_suite_test.go

+		Controller: config.Controller{
+			// see https://github.com/kubernetes-sigs/controller-runtime/issues/2937
+			// in short, our tests reuse the same name (reasonably so) and the controller-runtime
+			// package does not reset the stack of controller names between tests, so we disable
+			// the name validation here.
+			SkipNameValidation: ptr.To(true),
+		},


The comment calls out why this is needed, but this due to a recent c-r change that enforces stricter validation for controller names.

timflannagan · 2024-09-20T16:25:28Z

projects/gloo/api/v1/enterprise/options/extauth/v1/solo-kit.json

-  "gen_kube_types": true,
+  "gen_kube_types": false,


Moved this to solo-io#10079. For context, solo-kit was generating hack/*-codegen.sh bash scripts for these nested directories that were relevant, so toggling this off / removing this option helps us manage maintenance.

timflannagan · 2024-09-20T16:27:08Z

projects/gloo/pkg/api/v1/enterprise/options/rbac/rbac.pb.uniquehash.go

Just a quick note on these new uniquehash.go files. This is needed as we had a bug in GME's AccessPolicy caching that required us to introduce a new primitive in this library. See https://github.com/solo-io/gloo-mesh-enterprise/pull/17392 for more information. We aren't using this new method, but still wanted to provide context on these generated files as we're bumping skv2.

timflannagan · 2024-09-20T16:29:02Z

projects/gloo/pkg/api/v1/kube/client/clientset/versioned/typed/gloo.solo.io/v1/upstream.go

The diff in these client-gen generated files is a bit confusing. Basically, client-go had a series of improvements in 1.31 to help adopt generics and cut down on the amount of generated code for consumers of this library. The gentype package defines the common Get/List/etc. interfaces now.

timflannagan · 2024-09-20T16:32:09Z

projects/gloo/pkg/api/v1/kube/client/listers/gloo.solo.io/v1/upstream.go

Similar to the above comment about client-go generated code, client-go refactored the listers implementation to adopt a generics-based approach. See kubernetes/kubernetes#121574 for more information.

timflannagan · 2024-09-20T16:34:03Z

projects/gloo/pkg/plugins/kubernetes/plugin.go

I wanted to highlight this change in the sea of generated code changes. The primary change here is the removal of typecasting to corev1 listers which was causing the regression suite to fail. IMO, doing this is a violation of our own lister abstraction (that manages corev1 listers under-the-hood) and any net new issues with performance regressions could be handled as a follow-up in solo-kit.

timflannagan · 2024-09-20T16:34:41Z

test/e2e/aggregate_listener_test.go

+				EnableGatewayController: &wrappers.BoolValue{
+					Value: true,
+				},


Need to confirm with Tyler or Sam why this change is necessary.

Oh yes, let's chat about this, I have some context and questions

We followed up in Slack. We discussed how this is due to some unknown proto changes that affect boolean values to be overridden in our tests. This impacts just this test, because we require the EnableGatewayController (edge gw) to be true in Settings, but since we define some other values in the same struct, the default true value is not being respected, and instead the overriding empty value is being used so it is false.

Our plan is two-fold:

Keep this temporary solution to merge the large code. This way this PR doesn't go out of date

Immediately after, investigate what proto changes could lead to this and provide an explanation and fix

cc @timflannagan

Added this as a TODO on the parent issue as well.

sam-heilbron · 2024-09-23T18:24:14Z

Adding the work in progress label to prevent auto-merging, while we confirm this can be pulled into enterprise seamlessly

jmhbh

LGTM, just a small nit and a quick q. Thanks for working on this Tim!

jmhbh · 2024-09-24T00:41:18Z

.github/workflows/pr-kubernetes-tests.yaml

@@ -67,12 +67,12 @@ jobs:

        # September 16, 2024: 21 minutes
        - cluster-name: 'cluster-three'
-          go-test-args: '-v -timeout=25m'
+          go-test-args: '-v -timeout=30m'


just curious why the timeout bump is necessary?

@sam-heilbron Any idea on why this was necessary?

projects/gloo/pkg/plugins/kubernetes/plugin.go

jmhbh · 2024-09-24T01:06:56Z

Adding the work in progress label to prevent auto-merging, while we confirm this can be pulled into enterprise seamlessly

Created this branch k8s-1.31-bump to test the enterprise integration.

Signed-off-by: timflannagan <[email protected]>

timflannagan · 2024-09-24T21:51:06Z

Created this branch k8s-1.31-bump to test the enterprise integration.

Just saw this now. I opened solo-io#10069 earlier today to work through all the potential issues.

timflannagan and others added 2 commits September 19, 2024 18:18

*: Re-generate codegen

Loading
Loading status checks…

ef86d98

Signed-off-by: timflannagan <[email protected]>

timflannagan requested a review from a team as a code owner September 19, 2024 19:03

timflannagan added 2 commits September 19, 2024 19:43

Fix translator unit tests

45180c7

Signed-off-by: timflannagan <[email protected]>

Move changelog to correct release directory

Loading
Loading status checks…

9a3d1a9

Signed-off-by: timflannagan <[email protected]>

timflannagan closed this Sep 19, 2024

timflannagan reopened this Sep 19, 2024

timflannagan added 6 commits September 19, 2024 20:45

*: Bump to k8s 1.31.1

2159fde

Signed-off-by: timflannagan <[email protected]>

projects: Toggle gen_kube_types off for relevant projects

Loading
Loading status checks…

81fc1c3

Signed-off-by: timflannagan <[email protected]>

projects: Fix codegen

216f9ae

Signed-off-by: timflannagan <[email protected]>

Merge remote-tracking branch 'upstream/main' into chore/k8s-1-31

Loading
Loading status checks…

2eb95e6

Merge branch 'main' into chore/k8s-1-31

Loading
Loading status checks…

b78fab1

timflannagan commented Sep 20, 2024

View reviewed changes

changelog/v1.18.0-beta23/9683-k8s-1.31.yaml

@@ -0,0 +1,12 @@

changelog:

Copy link

Member Author

timflannagan Sep 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: Fix this changelog.

timflannagan commented Sep 20, 2024

View reviewed changes

artberger approved these changes Sep 23, 2024

View reviewed changes

Merge branch 'main' into chore/k8s-1-31

Loading
Loading status checks…

5fc1a2a

sam-heilbron added the work in progress label Sep 23, 2024

sam-heilbron approved these changes Sep 23, 2024

View reviewed changes

jmhbh approved these changes Sep 24, 2024

View reviewed changes

timflannagan added 4 commits September 24, 2024 19:03

*: Bump go-control-plane

a17eee3

Signed-off-by: timflannagan <[email protected]>

Merge remote-tracking branch 'origin/chore/k8s-1-31' into chore/k8s-1-31

f5e9d5e

plugins: Remove stale code comment

Loading
Loading status checks…

83c6c0b

Signed-off-by: timflannagan <[email protected]>

Merge remote-tracking branch 'upstream/main' into chore/k8s-1-31

Loading
Loading status checks…

fcf1045

timflannagan removed the work in progress label Sep 24, 2024

jmhbh approved these changes Sep 24, 2024

View reviewed changes

soloio-bulldozer bot merged commit c831171 into kgateway-dev:main Sep 24, 2024
18 checks passed

timflannagan deleted the chore/k8s-1-31 branch September 24, 2024 22:31

timflannagan mentioned this pull request Sep 27, 2024

[Epic] Bump k8s.io to v0.31.x across various repositories #9683

Closed

15 tasks

github-actions bot mentioned this pull request Nov 14, 2024

[Migrated] *: Bump k8s.io and controller-runtime dependencies solo-io/gloo#10069

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: Bump k8s.io and controller-runtime dependencies #10069

*: Bump k8s.io and controller-runtime dependencies #10069

timflannagan commented Sep 19, 2024 •

edited

Loading

solo-changelog-bot bot commented Sep 19, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024 •

edited

Loading

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

timflannagan Sep 20, 2024

sam-heilbron Sep 20, 2024

sam-heilbron Sep 20, 2024

timflannagan Sep 24, 2024

sam-heilbron commented Sep 23, 2024

jmhbh left a comment

jmhbh Sep 24, 2024

timflannagan Sep 24, 2024

jmhbh commented Sep 24, 2024

timflannagan commented Sep 24, 2024

*: Bump k8s.io and controller-runtime dependencies #10069

*: Bump k8s.io and controller-runtime dependencies #10069

Conversation

timflannagan commented Sep 19, 2024 • edited Loading

Description

Code changes

CI changes

Context

Notes for reviewers

Checklist:

solo-changelog-bot bot commented Sep 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timflannagan Sep 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sam-heilbron commented Sep 23, 2024

jmhbh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmhbh commented Sep 24, 2024

timflannagan commented Sep 24, 2024

timflannagan commented Sep 19, 2024 •

edited

Loading

timflannagan Sep 20, 2024 •

edited

Loading