ci: Use explicit templates for Windows infra #5906

jackfrancis · 2025-10-14T20:06:06Z

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

This PR makes 3 changes:

Rather than define cluster templates that can accommodate Windows infrastructure (and start w/ a zero scale capacity), we use explicit Windows templates to manage Windows-specific cluster scenarios. In practice these templates are for use during tests that create a single, static cluster config once. Overloading the templates to accommodate multiple operational use cases makes things unnecessarily complicated and can break tests if any part of the Windows surface area breaks.
Standardize to "true" being the significant value for the TEST_WINDOWS (and back-compat WINDOWS) environment variables that CI accommodates.
There are now new ("-windows") templates for all of the general purpose test templates that we use. Those template files will be used when TEST_WINDOWS=true is passed in at runtime.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

TODOs:

squashed commits
includes documentation
adds unit tests
cherry-pick candidate

Release note:

NONE

codecov · 2025-10-14T20:10:30Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 46.94%. Comparing base (e12a33a) to head (414e799).
⚠️ Report is 13 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #5906   +/-   ##
=======================================
  Coverage   46.94%   46.94%           
=======================================
  Files         279      279           
  Lines       29688    29688           
=======================================
  Hits        13937    13937           
  Misses      14938    14938           
  Partials      813      813

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

jackfrancis · 2025-10-14T23:20:09Z

/label tide/merge-method-squash

jackfrancis · 2025-10-14T23:28:45Z

/test pull-cluster-api-provider-azure-e2e-optional

jackfrancis · 2025-10-14T23:29:01Z

/test ls

k8s-ci-robot · 2025-10-14T23:29:04Z

@jackfrancis: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-provider-azure-apiversion-upgrade

/test pull-cluster-api-provider-azure-build

/test pull-cluster-api-provider-azure-ci-entrypoint

/test pull-cluster-api-provider-azure-e2e

/test pull-cluster-api-provider-azure-e2e-aks

/test pull-cluster-api-provider-azure-test

/test pull-cluster-api-provider-azure-verify

The following commands are available to trigger optional jobs:

/test pull-cluster-api-provider-azure-apidiff

/test pull-cluster-api-provider-azure-apiserver-ilb

/test pull-cluster-api-provider-azure-capi-e2e

/test pull-cluster-api-provider-azure-conformance

/test pull-cluster-api-provider-azure-conformance-azl3-with-ci-artifacts

/test pull-cluster-api-provider-azure-conformance-custom-builds

/test pull-cluster-api-provider-azure-conformance-dual-stack-with-ci-artifacts

/test pull-cluster-api-provider-azure-conformance-ipv6-with-ci-artifacts

/test pull-cluster-api-provider-azure-conformance-with-ci-artifacts

/test pull-cluster-api-provider-azure-conformance-with-ci-artifacts-dra

/test pull-cluster-api-provider-azure-dra-scalability

/test pull-cluster-api-provider-azure-e2e-optional

/test pull-cluster-api-provider-azure-e2e-workload-upgrade

/test pull-cluster-api-provider-azure-load-test-1k-dra-with-workload-custom-builds

/test pull-cluster-api-provider-azure-load-test-custom-builds

/test pull-cluster-api-provider-azure-load-test-dra-custom-builds

/test pull-cluster-api-provider-azure-load-test-dra-with-workload-custom-builds

/test pull-cluster-api-provider-azure-perf-test-apiserver-availability

/test pull-cluster-api-provider-azure-windows-custom-builds

/test pull-cluster-api-provider-azure-windows-with-ci-artifacts

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-provider-azure-apidiff

pull-cluster-api-provider-azure-apiversion-upgrade

pull-cluster-api-provider-azure-build

pull-cluster-api-provider-azure-ci-entrypoint

pull-cluster-api-provider-azure-conformance

pull-cluster-api-provider-azure-conformance-custom-builds

pull-cluster-api-provider-azure-conformance-dual-stack-with-ci-artifacts

pull-cluster-api-provider-azure-conformance-ipv6-with-ci-artifacts

pull-cluster-api-provider-azure-conformance-with-ci-artifacts

pull-cluster-api-provider-azure-conformance-with-ci-artifacts-dra

pull-cluster-api-provider-azure-e2e

pull-cluster-api-provider-azure-e2e-aks

pull-cluster-api-provider-azure-e2e-workload-upgrade

pull-cluster-api-provider-azure-test

pull-cluster-api-provider-azure-verify

In response to this:

/test ls

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jackfrancis · 2025-10-15T02:16:43Z

Not sure if this is a transient failure (is canadacentral not a region we push images to?)

  - lastTransitionTime: "2025-10-14T23:41:24Z"
    message: |
      scalesets failed to create or update. err: failed to create or update resource capz-e2e-zoh8q5-vmss/win-p-win (service: scalesets): PUT https://management.azure.com/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-e2e-zoh8q5-vmss/providers/Microsoft.Compute/virtualMachineScaleSets/win-p-win
      --------------------------------------------------------------------------------
      RESPONSE 404: 404 Not Found
      ERROR CODE: GalleryImageNotFound
      --------------------------------------------------------------------------------
      {
        "error": {
          "code": "GalleryImageNotFound",
          "message": "\"The gallery image /CommunityGalleries/ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019/Images/capi-win-2019-containerd/Versions/1.32.9 is not available in CanadaCentral region. Please contact image owner to replicate to this region, or change your requested region.\"",
          "target": "imageReference"
        }
      }
      --------------------------------------------------------------------------------
    reason: Failed
    severity: Error
    status: "False"
    type: BootstrapSucceeded
  image:
    computeGallery:
      gallery: ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019
      name: capi-win-2019-containerd
      version: 1.32.9
  infrastructureMachineKind: AzureMachinePoolMachine

Context: the failing E2E scenario in https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cluster-api-provider-azure/5906/pull-cluster-api-provider-azure-e2e/1978239364467003392

jackfrancis · 2025-10-15T02:16:47Z

/retest

jackfrancis · 2025-10-15T04:40:23Z

Not a flake:

  - lastTransitionTime: "2025-10-15T02:40:01Z"
    message: |
      scalesets failed to create or update. err: failed to create or update resource capz-e2e-hpjcyq-vmss/win-p-win (service: scalesets): PUT https://management.azure.com/subscriptions/46678f10-4bbb-447e-98e8-d2829589f2d8/resourceGroups/capz-e2e-hpjcyq-vmss/providers/Microsoft.Compute/virtualMachineScaleSets/win-p-win
      --------------------------------------------------------------------------------
      RESPONSE 404: 404 Not Found
      ERROR CODE: GalleryImageNotFound
      --------------------------------------------------------------------------------
      {
        "error": {
          "code": "GalleryImageNotFound",
          "message": "\"The gallery image /CommunityGalleries/ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019/Images/capi-win-2019-containerd/Versions/1.32.9 is not available in GermanyWestCentral region. Please contact image owner to replicate to this region, or change your requested region.\"",
          "target": "imageReference"
        }
      }
      --------------------------------------------------------------------------------
    reason: Failed
    severity: Error
    status: "False"
    type: BootstrapSucceeded
  image:
    computeGallery:
      gallery: ClusterAPI-f72ceb4f-5159-4c26-a0fe-2ea738f0d019
      name: capi-win-2019-containerd
      version: 1.32.9
  infrastructureMachineKind: AzureMachinePoolMachine

jackfrancis · 2025-10-15T04:40:54Z

/retest

jackfrancis · 2025-10-15T19:47:42Z

/cherry-pick release-1.21

k8s-infra-cherrypick-robot · 2025-10-15T19:47:45Z

@jackfrancis: once the present PR merges, I will cherry-pick it on top of release-1.21 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jackfrancis · 2025-10-15T20:06:34Z

/test pull-cluster-api-provider-azure-e2e-optional

jackfrancis · 2025-10-15T22:26:24Z

/test pull-cluster-api-provider-azure-e2e-windows

jackfrancis · 2025-10-15T23:48:49Z

/test pull-cluster-api-provider-azure-e2e-windows

k8s-ci-robot · 2025-10-16T00:59:48Z

@jackfrancis: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-e2e-optional	`da4f547`	link	false	`/test pull-cluster-api-provider-azure-e2e-optional`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

jackfrancis · 2025-10-16T18:40:13Z

/test pull-cluster-api-provider-azure-e2e-windows

mboersma

/lgtm

k8s-ci-robot · 2025-10-16T19:52:06Z

LGTM label has been added.

Git tree hash: 866ff0ffef1236dc6fcdf0f5580e50aa2d2e5821

jackfrancis · 2025-10-16T23:59:18Z

/approve

k8s-ci-robot · 2025-10-16T23:59:26Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jackfrancis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jackfrancis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jackfrancis · 2025-10-17T00:05:54Z

/cherry-pick release-1.21

k8s-infra-cherrypick-robot · 2025-10-17T00:05:56Z

@jackfrancis: once the present PR merges, I will cherry-pick it on top of release-1.21 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-infra-cherrypick-robot · 2025-10-17T01:37:17Z

@jackfrancis: #5906 failed to apply on top of branch "release-1.21":

Applying: ci: Use explicit templates for Windows infra
Using index info to reconstruct a base tree...
M	scripts/ci-entrypoint.sh
A	templates/test/ci/cluster-template-prow-ci-version-azl3.yaml
M	templates/test/ci/cluster-template-prow-ci-version-dra.yaml
M	templates/test/ci/cluster-template-prow-ci-version-dual-stack.yaml
M	templates/test/ci/cluster-template-prow-ci-version-md-and-mp.yaml
M	templates/test/ci/cluster-template-prow-ci-version.yaml
M	templates/test/ci/cluster-template-prow-machine-pool-ci-version.yaml
M	templates/test/ci/cluster-template-prow-machine-pool-flex.yaml
M	templates/test/ci/cluster-template-prow-machine-pool.yaml
M	templates/test/ci/cluster-template-prow-topology.yaml
M	templates/test/ci/cluster-template-prow.yaml
M	templates/test/dev/cluster-template-custom-builds-dra.yaml
M	templates/test/dev/cluster-template-custom-builds-load-dra.yaml
M	templates/test/dev/cluster-template-custom-builds-load.yaml
M	templates/test/dev/cluster-template-custom-builds-machine-pool-load-dra.yaml
M	templates/test/dev/cluster-template-custom-builds-machine-pool-load.yaml.git/rebase-apply/patch:8940: trailing whitespace.
    /var/lib/kubelet/credential-provider-config.yaml  
.git/rebase-apply/patch:9263: trailing whitespace.
      
.git/rebase-apply/patch:9297: trailing whitespace.
    /var/lib/kubelet/credential-provider-config.yaml  
.git/rebase-apply/patch:15677: trailing whitespace.
      
.git/rebase-apply/patch:15780: trailing whitespace.
      
warning: squelched 1 whitespace error
warning: 6 lines add whitespace errors.

M	templates/test/dev/cluster-template-custom-builds-machine-pool.yaml
M	templates/test/dev/cluster-template-custom-builds.yaml
M	test/e2e/helpers.go
Falling back to patching base and 3-way merge...
Auto-merging test/e2e/helpers.go
CONFLICT (content): Merge conflict in test/e2e/helpers.go
Auto-merging templates/test/dev/cluster-template-custom-builds.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-machine-pool.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds-machine-pool.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-machine-pool-load.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds-machine-pool-load.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-machine-pool-load-dra.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds-machine-pool-load-dra.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-load.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds-load.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-load-dra.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds-load-dra.yaml
Auto-merging templates/test/dev/cluster-template-custom-builds-dra.yaml
CONFLICT (content): Merge conflict in templates/test/dev/cluster-template-custom-builds-dra.yaml
Removing templates/test/ci/prow-topology/cni-resource-set.yaml
Auto-merging templates/test/ci/cluster-template-prow.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow.yaml
Auto-merging templates/test/ci/cluster-template-prow-topology.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-topology.yaml
Auto-merging templates/test/ci/cluster-template-prow-machine-pool.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-machine-pool.yaml
Auto-merging templates/test/ci/cluster-template-prow-machine-pool-flex.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-machine-pool-flex.yaml
Auto-merging templates/test/ci/cluster-template-prow-machine-pool-ci-version.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-machine-pool-ci-version.yaml
Auto-merging templates/test/ci/cluster-template-prow-ci-version.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-ci-version.yaml
Auto-merging templates/test/ci/cluster-template-prow-ci-version-md-and-mp.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-ci-version-md-and-mp.yaml
Auto-merging templates/test/ci/cluster-template-prow-ci-version-ipv6.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-ci-version-ipv6.yaml
Auto-merging templates/test/ci/cluster-template-prow-ci-version-dual-stack.yaml
Auto-merging templates/test/ci/cluster-template-prow-ci-version-dra.yaml
CONFLICT (content): Merge conflict in templates/test/ci/cluster-template-prow-ci-version-dra.yaml
Auto-merging scripts/ci-entrypoint.sh
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config advice.mergeConflict false"
Patch failed at 0001 ci: Use explicit templates for Windows infra

In response to this:

/cherry-pick release-1.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jackfrancis · 2025-10-17T19:21:38Z

/kind failing-test

ci: Use explicit templates for Windows infra

78351e8

github-project-automation bot added this to CAPZ Planning Oct 14, 2025

k8s-ci-robot added the release-note-none Denotes a PR that doesn't merit a release note. label Oct 14, 2025

github-project-automation bot moved this to Todo in CAPZ Planning Oct 14, 2025

k8s-ci-robot added kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Oct 14, 2025

k8s-ci-robot requested review from jsturtevant and marosset October 14, 2025 20:06

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Oct 14, 2025

jackfrancis added 2 commits October 14, 2025 14:41

omit Windows pool from ClusterClass test scenario

ccfd498

use new machine-pool-windows template for VMSS test

c176acc

k8s-ci-robot added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Oct 14, 2025

jackfrancis added 2 commits October 15, 2025 10:05

add a separate Windows test

841af61

non-Windows test scenario should not test Windows LB

da4f547

kubernetes-sigs deleted a comment from k8s-ci-robot Oct 15, 2025

jackfrancis mentioned this pull request Oct 15, 2025

capz: windows-specific presubmit runs kubernetes/test-infra#35712

Merged

use shorter “win” name to avoid API issues

414e799

jackfrancis mentioned this pull request Oct 16, 2025

capz: explicit TEST_WINDOWS=true for Windows test kubernetes/test-infra#35722

Merged

mboersma approved these changes Oct 16, 2025

View reviewed changes

k8s-ci-robot assigned mboersma Oct 16, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 16, 2025

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 16, 2025

k8s-ci-robot merged commit 275f9ee into kubernetes-sigs:main Oct 17, 2025
32 checks passed

k8s-ci-robot added this to the v1.22 milestone Oct 17, 2025

github-project-automation bot moved this from Todo to Done in CAPZ Planning Oct 17, 2025

jackfrancis mentioned this pull request Oct 17, 2025

[release-1.21] Use explicit templates for Windows infra #5923

Open

jackfrancis deleted the windows-templates-explicit branch October 17, 2025 19:21

k8s-ci-robot added the kind/failing-test Categorizes issue or PR as related to a consistently or frequently failing test. label Oct 17, 2025

ci: Use explicit templates for Windows infra #5906

ci: Use explicit templates for Windows infra #5906

Conversation

jackfrancis commented Oct 14, 2025

Uh oh!

codecov bot commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jackfrancis commented Oct 14, 2025

Uh oh!

jackfrancis commented Oct 14, 2025

Uh oh!

jackfrancis commented Oct 14, 2025

Uh oh!

k8s-ci-robot commented Oct 14, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

k8s-infra-cherrypick-robot commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

jackfrancis commented Oct 15, 2025

Uh oh!

k8s-ci-robot commented Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jackfrancis commented Oct 16, 2025

Uh oh!

mboersma left a comment

Choose a reason for hiding this comment

Uh oh!

k8s-ci-robot commented Oct 16, 2025

Uh oh!

jackfrancis commented Oct 16, 2025

Uh oh!

k8s-ci-robot commented Oct 16, 2025

Uh oh!

jackfrancis commented Oct 17, 2025

Uh oh!

k8s-infra-cherrypick-robot commented Oct 17, 2025

Uh oh!

Uh oh!

k8s-infra-cherrypick-robot commented Oct 17, 2025

Uh oh!

jackfrancis commented Oct 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Oct 14, 2025 •

edited

Loading

k8s-ci-robot commented Oct 16, 2025 •

edited

Loading