k8s 1.32 upgrade (CI to use 1.31)

[mythi]

No description provided.

[bpradipt]

/lgtm

[wainersm]

Hi @mythi !

I'm not sure about this bump. The CI for operator doesn't run functional tests but rather mere install/uninstall operations, coco's functional tests rely on Kata CI. In there, it's still using v1.30 for sev-snp and tdx (v1.28 for s390x) if I'm not mistaken.

The question is: shouldn't we keep same k8s version used in here and Kata CI? If yes, then we probably should bump the version in kata CI first.

@ryansavino @AdithyaKrishnan @fidencio @stevenhorsman

[mythi]

I'm OK to drop 5258d2d if that's the concern?

[wainersm]

Hi @mythi !

I'm OK to drop 5258d2d if that's the concern?

That's what I had in mind.

[mythi]

That's what I had in mind.

I dropped the commit. However, I feel we should always be moving to the latest released k8s version to find potential problems as soon as possible.

[ldoktor]

Being a newbie here but I also think the upstream projects should try consuming the latest and greatest bits. Downstream should verify and patch things in case something gets broken, but upstream should move fast and break stuff, that's the purpose of upstream, isn't it?

As for the functional testing, I don't think the operator requires much but perhaps it'd be nice to bring back at least very simple scenario (the testing takes 20m so adding basic functional test shouldn't increase that significantly)

[wainersm]

That's what I had in mind.

I dropped the commit. However, I feel we should always be moving to the latest released k8s version to find potential problems as soon as possible.

I agree with you. I'm just concern on doing it right before a release and I think we should start bumping on kata CI side (due the lack of functional tests I mentioned). I will reach to maintainers of jobs on kata CI to see if they can upgrade to k8s 1.31.

[wainersm]

Being a newbie here but I also think the upstream projects should try consuming the latest and greatest bits. Downstream should verify and patch things in case something gets broken, but upstream should move fast and break stuff, that's the purpose of upstream, isn't it?

Agree.

As for the functional testing, I don't think the operator requires much but perhaps it'd be nice to bring back at least very simple scenario (the testing takes 20m so adding basic functional test shouldn't increase that significantly)

#446 is a tentative to introduce some basic function tests back. Unfortunately Gabriela switched to another project, so she won't be able to keep working on it.

[wainersm]

Thanks @mythi !

[wainersm]

I see the error in sev and snp-sev jobs:

::info:: Bring up the test cluster
+ ./cluster/up.sh
W1125 13:39:21.695747 3509546 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
[init] Using Kubernetes version: v1.30.1
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.31.1" Control plane version: "1.30.1"
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

The previous version of this PR had 1.31 installed then latest reverted back to 1.30. Maybe CI didn't wipe out kubelet from the node properly?

I'm re-running the jobs to see if they pass...

@AdithyaKrishnan @ryansavino could you have a look if they don't pass?

[mythi]

FYI, I added af08f6f back.

[AdithyaKrishnan]

I see the error in sev and snp-sev jobs:

::info:: Bring up the test cluster
+ ./cluster/up.sh
W1125 13:39:21.695747 3509546 common.go:101] your configuration file uses a deprecated API spec: "kubeadm.k8s.io/v1beta3" (kind: "ClusterConfiguration"). Please use 'kubeadm config migrate --old-config old.yaml --new-config new.yaml', which will write the new, similar spec using a newer API version.
[init] Using Kubernetes version: v1.30.1
[preflight] Running pre-flight checks
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR KubeletVersion]: the kubelet version is higher than the control plane version. This is not a supported version skew and may lead to a malfunctional cluster. Kubelet version: "1.31.1" Control plane version: "1.30.1"
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

The previous version of this PR had 1.31 installed then latest reverted back to 1.30. Maybe CI didn't wipe out kubelet from the node properly?

I'm re-running the jobs to see if they pass...

@AdithyaKrishnan @ryansavino could you have a look if they don't pass?

Fixed @wainersm @ryansavino

[mythi]

It looks the sev-snp cleanup isn't working well. The latest run suggests that a cluster is still running?

[wainersm]

Hi @mythi , regarding the bump of k8s version, yesterday on CI meeting we talked about a policy. On Kata CI, the non-TEE job, is running in AKS with unpinned version; meaning it will be using the latest available version according to AKS's policy which is getting N-1. For example, currently on the east-us region N = 1.30, so it's installed are using 1.29. While we won't be on latest and greatest upstream k8s version, we agreed that it will be sufficiently close and enough for our testing purposes. On operator side, we will be free to pick up the latest upstream k8s version + we plan to introduce some functional (smoke) tests to help on finding any breakage.

[mythi]

I re-requested a review from @wainersm as I bumped to 1.32 k8s version for the dependencies. CI is still at 1.31.