Repository navigation
ci: fix the Windows e2e job on current GitHub runners - #1464
Merged
Merged
Conversation
The nightly e2e run has failed on Windows every day since early
September 2025. Two runner image changes broke the k3s server that
knicknic/temp-kubernetes-ci starts:
1. windows-latest moved to Windows Server 2025, which no longer ships
wmic, so the kubelet exits with:
exec: "wmic": executable file not found in %PATH%
Pin the job to windows-2022, which still has it.
2. The runner images now ship Docker 29, which rejects the Docker API
version (1.38) used by that k3s build:
client version 1.38 is too old. Minimum supported API version is 1.40
Set DOCKER_MIN_API_VERSION=1.38 for the Docker service and restart it
before starting k3s.
Fixes KusionStack#1237
Co-authored-by: Iamrushabhshahh <rushabhshah842@gmail.com>
Signed-off-by: Iamrushabhshahh <rushabhshah842@gmail.com>
Contributor
|
Great detective work! Pinning to windows-2022 plus the Docker API tweak is a pragmatic fix. Could we open a follow-up for when windows-2022 is retired? LGTM. |
This was referenced Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
/kind bug
What this PR does / why we need it:
What is broken
The nightly
e2e testworkflow has been red for more than a year. Onlye2e test on windowsfails;e2e test on linuxpasses. The last green scheduled run was on 2025-08-31 (https://github.com/KusionStack/kusion/actions/runs/17355638799). None of the 407 scheduled runs since 2025-09-01 has passed (the GitHub API returns 0 runs withstatus=success). The Windows job also fails on every PR, including the Dependabot PRs #1452, #1454, #1457, #1459 and #1460.Latest failing run (2026-10-14): https://github.com/KusionStack/kusion/actions/runs/38548649405/job/115800902226
Root cause
The
setup K3sstep usesknicknic/temp-kubernetes-ci@v1. The action has not changed since January 2023. It downloads an old Windows build of k3s that uses Docker as its container runtime, and starts it in the background. The step reports success even when k3s dies, so the cluster never comes up (dial tcp 127.0.0.1:6443: connectex: No connection could be made). The spec that needs the cluster then times out:Two runner image changes broke that k3s build.
windows-latestis now Windows Server 2025 (Windows-latest workflows will use Windows Server 2025 image in GH actions/runner-images#12677, rolled out 2025-09-02 to 2025-09-30). The current run logsImage: windows-2025-vs2026. Windows Server 2025 does not shipwmic, and the kubelet exits:wmicwas back, but the kubelet still exited:The log also shows
No MSFT_NetIPAddress objects found with property 'InterfaceAlias' equal to 'Ethernet'from the action script, because the adapter is now calledEthernet 3orEthernet 4. The action does not pass that value to k3s, so this error does no harm. The passing run below still shows it.What this PR changes
Only
.github/workflows/e2e-test.yml, and only thee2e-windowsjob:runs-on: windows-latestbecomesruns-on: windows-2022, with a comment that links CI: Window E2E test failed probabilistically #1237.setup K3ssetsDOCKER_MIN_API_VERSION=1.38in the Docker service environment (HKLM:\SYSTEM\CurrentControlSet\Services\docker, valueEnvironment), restarts Docker and printsdocker version. A machine-wide environment variable would not work here, because Windows services only read those at boot.The Linux job and the other steps are unchanged.
Why this option
DOCKER_MIN_API_VERSIONas the supported way to accept older clients. It only affects this throwaway runner.DISM /Online /Add-Capability /CapabilityName:WMIC~~~~). This downloads from Windows Update on every run and still needs the Docker fix. I did not try it.knicknic/temp-kubernetes-ci. This is the right long-term fix. There is no maintained drop-in action that runs Kubernetes natively on a Windows runner, though. Choosing between k3d or kind on WSL2, or dropping the Kubernetes-backed spec on Windows, is a design decision for the maintainers, so I kept it out of this fix.How I tested it
I cannot run Windows CI locally, so I ran the workflow on my fork. I pushed this branch with one temporary commit, which is not part of this PR, so that
e2e-test.ymlalso runs on push totmp/**branches.e2e test on windowspassed:Ran 4 of 4 Specs ... 4 Passed). So the same run now shows both jobs green.The branch applies cleanly on
main. It also merges cleanly with the Go 1.26 CI PR, which edits other lines ofe2e-test.yml.Risks and follow-ups
DOCKER_MIN_API_VERSION, the same kubelet error will come back.D:\a\kusion\kusion\bin. windows-2022 has aD:drive, so this works for now. It is one more thing to change when the job moves to a newer image.Which issue(s) this PR fixes:
Related: #1237. Per the PR template I have not used
Fixes, because this is a failing-test fix. #1237 can be closed once the nightly run is green onmain.Special notes for your reviewer:
@ruquanzhao, could you review this when you get a chance? cc @SparkYuan, since you opened #1237. The change only touches
.github/workflows/e2e-test.yml, and it merges cleanly with the Go 1.26 PR.Does this PR introduce a user-facing change?
Additional documentation e.g., design docs, usage docs, etc.: