ci: IL2CPP standalone legs fail because cl.exe is missing on the Windows runner

[wallstop]

Problem

Every standalone (IL2CPP) leg on the self-hosted Windows runner is failing because the MSVC C++ compiler is not present at the path Unity detected. This is a host toolchain outage, not a code regression, and it affects master as much as any pull request.

Evidence

Run 30665529494. Two standalone legs failed with the identical error, from two different Unity versions:

Building Library\Bee\artifacts\WinPlayerBuildProgram\qc86f\so38ugmnna5n.obj failed with output:
'"C:/Program Files/Microsoft Visual Studio/18/Community/VC/Tools/MSVC/14.51.36231/bin/Hostx64/x64/cl.exe"'
is not recognized as an internal or external command, operable program or batch file.
...
Player build failed
TestLaunchFailedException: Player build failed

Unity resolved the toolchain path and put it on PATH:

PATH=C:\Program Files\Microsoft Visual Studio\18\Community\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x64;C:\Program Files (x86)\Windows Kits\10\bin\10.0.28000.0\x64

so the version directory exists and was discovered, but cl.exe under bin\Hostx64\x64 is not executable. That is the signature of a partially completed or in-flight Visual Studio update, not of a missing install.

leg	result
Unity 2021.3.45f1 standalone	failure -- cl.exe not recognized
Unity 2022.3.45f1 standalone	failure -- cl.exe not recognized
Unity 2021.3/2022.3 editmode + playmode	success (Mono; no C++ toolchain needed)

It is a regression in the host, not the repository

The last full Unity Tests run on master, 30658003577 (ac1af7de, 19:09Z), had all three standalone legs green. The failures above began around 21:19Z. Nothing in between changed scripts/unity/run-ci-tests.ps1, any C# source, or any build configuration.

The editmode and playmode legs passing in the same runs is the clean discriminator: they run under Mono and never invoke cl.exe.

The prerequisite gate does not cover this

.github/actions/assert-unity-host-prereqs checks the Visual C++ runtime redistributables (MSVCP100.dll, VCRUNTIME140.dll, ...), which is what the Editor needs in order to launch. It does not check the Visual C++ build tools, which is what IL2CPP needs in order to compile a player.

The consequence is the expensive failure mode: the leg acquires the organization Unity build lock, consumes a scarce two-seat Unity licence, builds for roughly twenty minutes, and only then discovers the compiler is absent. A prerequisite check would fail it in seconds, before the lock is taken.

Suggested fix

1. Host, immediately: repair or complete the Visual Studio C++ workload on the affected runner so …\VC\Tools\MSVC\<version>\bin\Hostx64\x64\cl.exe exists and runs.
2. Repository, to prevent recurrence: extend assert-unity-host-prereqs to assert an executable cl.exe for the newest installed MSVC toolset (resolved with vswhere, which is the supported discovery mechanism), on the same fail-fast footing as the redistributable checks. Only the standalone legs need it, so it can be gated on matrix.test-mode == 'standalone' to avoid taxing editmode and playmode.
3. Consider having the runner opt out of unattended Visual Studio updates, since an update window silently removes the compiler mid-run.

Acceptance criteria

• A runner missing an executable cl.exe fails before Acquire organization Unity lock, not twenty minutes into an IL2CPP build.
• The failure message names the missing compiler path, rather than surfacing as Player build failed / tests did not run.
• Editmode and playmode legs are unaffected by the new check.

Related

• scripts/unity/bootstrap-windows-runner.ps1 -- the existing remediation for the runtime redistributables; the build tools are the same class of host prerequisite and belong alongside it.

[wallstop]

Complete matrix result: the split is exact

Run 30665529494 finished. All nine matrix legs across three Unity versions, and the outcome divides perfectly on whether the leg needs a C++ compiler:

leg	backend	needs cl.exe	result
2021.3.45f1 editmode	Mono	no	success
2021.3.45f1 playmode	Mono	no	success
2022.3.45f1 editmode	Mono	no	success
2022.3.45f1 playmode	Mono	no	success
6000.3.16f1 editmode	Mono	no	success
6000.3.16f1 playmode	Mono	no	success
2021.3.45f1 standalone	IL2CPP	yes	failure
2022.3.45f1 standalone	IL2CPP	yes	failure
6000.3.16f1 standalone	IL2CPP	yes	failure

Six for six on Mono, zero for three on IL2CPP. All three IL2CPP legs carry the same two lines:

'"...\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x64\cl.exe"' is not recognized as an internal or external command
Player build failed

A code regression cannot produce that shape -- the same C#, the same test assemblies, and the same harness pass under Mono in the very same runs. Only the native compile step is broken, and it is broken identically on every Unity version, which points at the shared host toolchain rather than at anything version-specific.

Worth noting for whoever repairs the host: the Unity lifecycle behaved correctly throughout. Every failing leg still returned its licence and released the organization build lock (Return Unity license: success, Require confirmed Unity cleanup: success), so no seat was leaked by these failures.

This blocks Unity CI Success on master and on every open pull request until the compiler is restored on the runner.

[wallstop]

Ruled out: Library cache poisoning

Worth recording so nobody spends time on it. The obvious repo-side theory is that the cached Library/ (which includes Library/Bee/artifacts, where the failing compile command lines live) was built against the old toolchain and its cache key does not include the MSVC version -- so a host toolchain change would not invalidate it.

That is not what happened. The cache key resolves to the same hash for the passing and failing runs:

6216660584  last_accessed 2026-07-31T21:18:42Z  Library-Windows-X64-2021.3.45f1-standalone-912f9444...
6217128118  last_accessed 2026-07-31T21:54:07Z  Library-Windows-X64-2022.3.45f1-standalone-912f9444...
6217556552  last_accessed 2026-07-31T21:59:55Z  Library-Windows-X64-6000.3.16f1-standalone-912f9444...

The key hashes package.json, Runtime/**, Editor/**, Tests/**, run-ci-tests.ps1, asmdef-discovery.js, and the compute-assemblies action. PR #331 touches none of them, so it restored the same entry that master's green run at 19:09Z used, and actions/cache does not overwrite an existing key.

Same cache, same code, opposite outcome. The variable is the host, between 19:09Z and 21:18Z. Busting the cache will not help, and neither will re-running.

That said, the cache key genuinely should carry the toolchain identity -- a Bee graph is toolchain-specific and a cache that outlives the compiler it was built against is a real (if separate) hazard. Folding the resolved MSVC toolset version into the key is worth doing alongside the cl.exe prerequisite check, since one vswhere call yields both.

The fix remains a host action: restore the Visual Studio C++ workload on the runner.

[wallstop]

The runner has recovered; the outage was a transient update window

Run 30669141848, same branch, same code, roughly an hour after the failures:

success  Unity 2021.3.45f1 standalone

That is the same leg, with the same Library cache entry, that failed twice with cl.exe is not recognized. Nothing in the repository changed between the failing and passing runs that touches the Unity build path -- the only commit in between logs a summary line in stuck-job-watchdog.yml, which no Unity leg reads.

This confirms the diagnosis in the opening report: the MSVC toolchain was temporarily absent, which is what a Visual Studio update window looks like from inside a build. It was not a missing install, and it was not a repository regression.

This issue should stay open

The outage self-resolving does not remove the gap it exposed:

1. The prerequisite gate does not cover the C++ build tools. During the window, three IL2CPP legs each acquired the organization Unity build lock, consumed one of two licence seats, built for roughly twenty minutes, and only then discovered the compiler was gone. A vswhere-based assertion that cl.exe is executable would have failed them in seconds, before the lock.
2. The window will recur. Unattended Visual Studio updates are the normal state of a developer-configured Windows host. The next one lands in the middle of another run.
3. The failure surface was misleading. Two legs reported Player build failed, the third reported a player crashing at startup with CreateDirectory '' failed. Neither names the real cause; an operator reading either would start in the wrong place. The 2021.3 leg looked like a player bug, which is what sent the first investigation down a blind alley.

The recurrence fix in the opening report stands, and is now better motivated: it converts a twenty-minute lock-holding failure with a misleading message into a fast one that names the missing compiler.

Suggested severity change: this is no longer an active outage, but it remains a real gap in the prerequisite gate.

[wallstop]

Correction: nothing recovered. It is DAD-MACHINE.

My earlier comment claiming the runner recovered was wrong, and I should not have concluded it from a passing run alone. The passing runs simply landed on the other host.

Run 30672255858 is the decisive one, because two standalone legs of the same run disagreed:

run	leg	runner	result
30672255858	2021.3 standalone	DAD-MACHINE	failure -- cl.exe not recognized
30672255858	2022.3 standalone	ELI-MACHINE	success
30665529494	all three standalone	DAD-MACHINE	0/3
30669141848	all three standalone	ELI-MACHINE	3/3
30670645032	all three standalone	ELI-MACHINE	3/3

Every failure is DAD-MACHINE. Every success is ELI-MACHINE. Across four runs, three Unity versions, and both a completely-red and a completely-green matrix, the correlation has no exceptions. The variable is the host, not the hour.

So the diagnosis is narrower and more actionable than "a transient Visual Studio update window":

DAD-MACHINE is missing an executable cl.exe at the MSVC toolset path Unity discovers there (...\VC\Tools\MSVC\14.51.36231\bin\Hostx64\x64\). ELI-MACHINE is fine. Whether that is an interrupted update, a partially removed toolset, or a workload that was never installed is a question for the host.

Why this looks intermittent from the outside

unity-tests.yml runs max-parallel: 1 across a two-runner pool, so which host takes a given leg is effectively arbitrary. A matrix therefore comes back fully green, fully red, or mixed depending purely on scheduling -- which reads as flakiness and is why the first pass through this looked like a time-bounded outage. It is not: it is deterministic per host.

What this changes

1. The fix is a targeted repair of one machine, not a wait-and-see.
2. The prerequisite gate matters more, not less. A vswhere-based cl.exe assertion would fail a DAD-MACHINE standalone leg in seconds and name the host, instead of consuming a Unity seat and twenty minutes of IL2CPP build before dying with Player build failed. It also makes the per-host nature obvious in the annotation rather than requiring someone to correlate runner_name across four runs.
3. Until the host is repaired, roughly half of all standalone legs will fail, on master and on every pull request, purely by scheduling.

Apologies for the earlier premature all-clear.