Skip to content

ci: fix the Windows e2e job on current GitHub runners - #1464

Merged
elliotxx merged 1 commit into
KusionStack:mainfrom
Iamrushabhshahh:ci/windows-e2e-k3s
Oct 15, 2026
Merged

elliotxx merged 1 commit into
KusionStack:mainfrom
Iamrushabhshahh:ci/windows-e2e-k3s

Conversation

@Iamrushabhshahh

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind bug

What this PR does / why we need it:

What is broken

The nightly e2e test workflow has been red for more than a year. Only e2e test on windows fails; e2e test on linux passes. The last green scheduled run was on 2025-08-31 (https://github.com/KusionStack/kusion/actions/runs/17355638799). None of the 407 scheduled runs since 2025-09-01 has passed (the GitHub API returns 0 runs with status=success). The Windows job also fails on every PR, including the Dependabot PRs #1452, #1454, #1457, #1459 and #1460.

Latest failing run (2026-10-14): https://github.com/KusionStack/kusion/actions/runs/38548649405/job/115800902226

Root cause

The setup K3s step uses knicknic/temp-kubernetes-ci@v1. The action has not changed since January 2023. It downloads an old Windows build of k3s that uses Docker as its container runtime, and starts it in the background. The step reports success even when k3s dies, so the cluster never comes up (dial tcp 127.0.0.1:6443: connectex: No connection could be made). The spec that needs the cluster then times out:

[FAILED] Timed out after 900.009s.
[FAIL] kusion Runtime Commands [It] apply and destroy
FAIL! -- 3 Passed | 1 Failed | 0 Pending | 0 Skipped

Two runner image changes broke that k3s build.

  1. windows-latest is now Windows Server 2025 (Windows-latest workflows will use Windows Server 2025 image in GH actions/runner-images#12677, rolled out 2025-09-02 to 2025-09-30). The current run logs Image: windows-2025-vs2026. Windows Server 2025 does not ship wmic, and the kubelet exits:
    F1014 10:20:19.310782    6636 server.go:270] failed to run Kubelet: exec: "wmic": executable file not found in %PATH%
    
  2. The images now ship Docker 29 (29.7.2 on windows-2022 today). Docker 29 rejects the API version that this k3s build uses. I found this when I pinned the job to windows-2022 on my fork. wmic was back, but the kubelet still exited:
    F1014 21:31:45.433235    7552 server.go:270] failed to run Kubelet: failed to get docker version: Error response from daemon: client version 1.38 is too old. Minimum supported API version is 1.40, please upgrade your client to a newer v…
    
    (run https://github.com/Iamrushabhshahh/kusion/actions/runs/38640784571)

The log also shows No MSFT_NetIPAddress objects found with property 'InterfaceAlias' equal to 'Ethernet' from the action script, because the adapter is now called Ethernet 3 or Ethernet 4. The action does not pass that value to k3s, so this error does no harm. The passing run below still shows it.

What this PR changes

Only .github/workflows/e2e-test.yml, and only the e2e-windows job:

  • runs-on: windows-latest becomes runs-on: windows-2022, with a comment that links CI: Window E2E test failed probabilistically #1237.
  • A new step before setup K3s sets DOCKER_MIN_API_VERSION=1.38 in the Docker service environment (HKLM:\SYSTEM\CurrentControlSet\Services\docker, value Environment), restarts Docker and prints docker version. A machine-wide environment variable would not work here, because Windows services only read those at boot.

The Linux job and the other steps are unchanged.

Why this option

  • Pin to windows-2022 and lower Docker's minimum API version (chosen). Both changes are needed, and together they are the smallest change that makes the job pass. Docker documents DOCKER_MIN_API_VERSION as the supported way to accept older clients. It only affects this throwaway runner.
  • Stay on Windows Server 2025 and install WMIC as a Feature on Demand (DISM /Online /Add-Capability /CapabilityName:WMIC~~~~). This downloads from Windows Update on every run and still needs the Docker fix. I did not try it.
  • Replace knicknic/temp-kubernetes-ci. This is the right long-term fix. There is no maintained drop-in action that runs Kubernetes natively on a Windows runner, though. Choosing between k3d or kind on WSL2, or dropping the Kubernetes-backed spec on Windows, is a design decision for the maintainers, so I kept it out of this fix.

How I tested it

I cannot run Windows CI locally, so I ran the workflow on my fork. I pushed this branch with one temporary commit, which is not part of this PR, so that e2e-test.yml also runs on push to tmp/** branches.

  • windows-2022 pin only: https://github.com/Iamrushabhshahh/kusion/actions/runs/38640784571. Still failed, with the Docker API error above. This is how I found the second cause.
  • windows-2022 pin plus the Docker step (the YAML in this PR): https://github.com/Iamrushabhshahh/kusion/actions/runs/38644177266. e2e test on windows passed:
    Server: Docker Engine - Community
     Engine:
      Version:          29.7.2
      API version:      1.55 (minimum version 1.38)
    ...
    Ran 4 of 4 Specs in 29.630 seconds
    SUCCESS! -- 4 Passed | 0 Failed | 0 Pending | 0 Skipped
    
    In that run the Linux job first failed on a module download error, which is unrelated to this change. I re-ran just that job and it passed (Ran 4 of 4 Specs ... 4 Passed). So the same run now shows both jobs green.

The branch applies cleanly on main. It also merges cleanly with the Go 1.26 CI PR, which edits other lines of e2e-test.yml.

Risks and follow-ups

  • This keeps an unmaintained action alive. The next runner image change could break it again.
  • windows-2022 has no deprecation notice today. GitHub supports at most two GA Windows Server images at a time, and Windows Server 2025 is now the default. So windows-2022 is the next image to be retired once a newer Windows Server image goes GA. When GitHub announces that, the k3s setup must be replaced. I suggest keeping a follow-up issue open for that.
  • If a future Docker release drops support for API 1.38 even with DOCKER_MIN_API_VERSION, the same kubelet error will come back.
  • The job hard-codes D:\a\kusion\kusion\bin. windows-2022 has a D: drive, so this works for now. It is one more thing to change when the job moves to a newer image.

Which issue(s) this PR fixes:

Related: #1237. Per the PR template I have not used Fixes, because this is a failing-test fix. #1237 can be closed once the nightly run is green on main.

Special notes for your reviewer:

@ruquanzhao, could you review this when you get a chance? cc @SparkYuan, since you opened #1237. The change only touches .github/workflows/e2e-test.yml, and it merges cleanly with the Go 1.26 PR.

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., design docs, usage docs, etc.:


The nightly e2e run has failed on Windows every day since early
September 2025. Two runner image changes broke the k3s server that
knicknic/temp-kubernetes-ci starts:

1. windows-latest moved to Windows Server 2025, which no longer ships
   wmic, so the kubelet exits with:
     exec: "wmic": executable file not found in %PATH%
   Pin the job to windows-2022, which still has it.

2. The runner images now ship Docker 29, which rejects the Docker API
   version (1.38) used by that k3s build:
     client version 1.38 is too old. Minimum supported API version is 1.40
   Set DOCKER_MIN_API_VERSION=1.38 for the Docker service and restart it
   before starting k3s.

Fixes KusionStack#1237

Co-authored-by: Iamrushabhshahh <rushabhshah842@gmail.com>
Signed-off-by: Iamrushabhshahh <rushabhshah842@gmail.com>
@elliotxx

Copy link
Copy Markdown
Contributor

Great detective work! Pinning to windows-2022 plus the Docker API tweak is a pragmatic fix. Could we open a follow-up for when windows-2022 is retired? LGTM.

@elliotxx elliotxx left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@elliotxx
elliotxx merged commit 8569576 into KusionStack:main Oct 15, 2026
8 checks passed
@github-actions github-actions Bot locked and limited conversation to collaborators Oct 15, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants