[Epic] Bump k8s.io to v0.31.x across various repositories

[stevenctl]

Gloo Edge Product

Open Source

Gloo Edge Version

1.18

Is your feature request related to a problem? Please describe.

I want Gloo Edge to use the latest stable Kubernetes version

Tasklist

Core tasks:

### Tasks
- [x] Update other misc. dependencies like solo-kit, skv2, etc.
- [x] Merge https://github.com/solo-io/gloo/pull/10069
- [x] Release https://github.com/solo-io/gloo/releases/tag/v1.18.0-beta23
- [x] Merge https://github.com/solo-io/solo-apis/pull/1313
- [x] Merge https://github.com/solo-io/solo-projects/pull/6959

Nice-to-haves:

- [ ] https://github.com/solo-io/solo-projects/issues/6976
- [ ] https://github.com/solo-io/solo-projects/issues/6975
- [ ] https://github.com/solo-io/gloo/issues/10129
- [ ] https://github.com/solo-io/solo-kit/issues/566

---

Previous issue description:

High Level Tasks

• Walk through the Kubernetes release notes, understanding what has changed since K8s 1.29 (https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#changelog-since-v1290)
• Update individual repos. Use previous upgrades as inspiration for the order of libraries
• Work with @solo-io/solo-docs to update our support matrix and any other relevant user-facing docs that are impacted by this change

Walk through the Kubernetes release notes, understanding what has changed since K8s 1.29

https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.30.md#changelog-since-v1290

I don't see any horribly concerning behavior changes, although there is a lot of stuff done to improve JWTAuthenticators. I don't think this impacts us. There aren't any concerning top-level API changes.

Go API changes are mostly superficial:

• controller-runtime: leaderelection.SwitchMetric is now leaderelection.LeaderMetric and requires an additional SlowpathExercised handler.
• controller-runtime: some params to Watch moved to source.Kind (already handled in skv2, the only place I see it affecting our code)

Update Individual Repos

• cloud-builders: https://github.com/solo-io/cloud-builders/pull/124
• go-utils: TODO determine if necessary
• k8s-utils: bump k8s deps for 1.30 solo-io/k8s-utils#52
• solo-kit: bump k8s for 1.30 solo-io/solo-kit#557
• skv2 chore: update k8s dependencies solo-io/skv2#560
  • released as v0.40.4
  • this was done in May, bumped to k8s 0.30.0, dependent repos should be ok to use later patch versions
• rate-limiter
• solo-apis
• gloo: [Migrated] bump k8s dependencies for 1.30 solo-io/gloo#9612
• caching-service
• external-apis
• ext-auth-service
• solo-projects

Additional Context

solo-io#8978 [ ]

Additional Context

solo-io#8978

┆Issue is synchronized with this Asana task by Unito

[stevenctl]

Istio is already on 1.31 - should we catch up istio/istio#52661

[tjons]

After discussion, we are going to try to upgrade from 1.29 -> 1.31 as we would like to allow the changes to soak as long as possible before release. There is an open discussion around whether to include go 1.23 as well to knock out all the upgrades in one pass, I anticipate we will make that decision by tomorrow.

For today, I've documented information about the 1.30 -> 1.31 release:
The 1.31 Changelog is here: https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.31.md#changelog-since-v1300. There are some interface changes for custom scheduler plugins (which does not impact gloo), as well as the removal of the --keep-terminated-pod-volumes kubelet flag. This should not affect gloo either.

I will add any discovered issues as I work through the upgrade process in this github issue.

To-do list with individual repos

• go-utils - Update dependencies where possible and move to go1.23 solo-io/go-utils#531, released as v0.26.0
• cloud-builders - https://github.com/solo-io/cloud-builders/pull/126, released as v0.10.0, and https://github.com/solo-io/cloud-builders/pull/127, released as v0.10.1
• k8s-utils - go 1.23/k8s 1.31 solo-io/k8s-utils#53, released as v0.8.0
• solo-kit - k8s 1.31/go1.23 solo-io/solo-kit#560, released as v0.36.0
• skv2 - Kubernetes 1.31 and go1.23 solo-io/skv2#584, released as v0.41.0
• rate-limiter - https://github.com/solo-io/rate-limiter/pull/246, released as v0.13.0
• solo-apis
• gloo - [Migrated] Upgrade kubernetes dependencies to 1.31 and golang to 1.23 solo-io/gloo#10026, WIP
• caching-service - https://github.com/solo-io/caching-service/pull/28, released as v0.8.0
• external-apis - kubernetes 1.31 and istio bump solo-io/external-apis#47, released as v0.3.0
• ext-auth-service - https://github.com/solo-io/ext-auth-service/pull/792 and https://github.com/solo-io/ext-auth-service/pull/793, released as v0.61.0
• solo-projects

[tjons]

After discussion, we will complete solo-io#9819 in parallel, upgrading to go1.23 while upgrading to k8s 1.31. I anticipate significant time savings from doing this and expect minimal issues.

[timflannagan]

@tjons Just a heads up that bumping controller-runtime to 0.18+ will break our builds. I decided to replace pin that dependency in solo-io#9933. AFAIK, we're manually vendoring a package from c-r for leader election metrics (and c-r manually depends that package from client-go), so it's not a ton of leg work to get builds back online.

[timflannagan]

Just quickly tracking these TODOs to avoid losing the context while things continue to progress:

• Refactor this issue to be an epic?
• Bump rate-limiter to unblock RFE team
• Bump go-control-plane & projects/envoyinit/hack/filter_types/filter_types.gen.go
  • Handle any ext_proc plugin changes with updated go-control-plane versions. Note, this is likely downstream specific and not OSS facing
• Investigate the sizeCache changes with updated protobuf dependency. We needed to change unit tests that were previously using To(Equal(...)) gomega matcher semantics and needed to be updated to our custom matcher library or changed to To(BeEquivalentTo(...)) to avoid a deep equality check
  • Previously sizeCache was 0 and the newer protobuf dependency had sizeCache set to 1 (or vice-versa)
• https://github.com/solo-io/solo-projects/issues/6976
• https://github.com/solo-io/solo-projects/issues/6975

[timflannagan]

Going to close this out now that all the repos have been updated. I created issues for any follow-up work that needs to be tracked in the short term.