Bazel binaries built with the latest MSVC version (14.40.33807) crash with older vcruntime

[fmeum]

Description of the bug:

In GitHub Actions on a Windows runner, Bazelisk fails to launch Bazel 7.2.0rc3, while 7.2.0rc2 on Windows and 7.2.0rc3 on other OSes work just fine (see fmeum/buildozer#15).

The log output from the runner:

2024-06-06T07:15:48.7574869Z ##[group]Run bazelisk run //:run_buildozer
2024-06-06T07:15:48.7575484Z �[36;1mbazelisk run //:run_buildozer�[0m
2024-06-06T07:15:48.7575868Z �[36;1mbazelisk test //...�[0m
2024-06-06T07:15:48.7611914Z shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
2024-06-06T07:15:48.7612417Z env:
2024-06-06T07:15:48.7612670Z   USE_BAZEL_VERSION: 7.2.0rc3
2024-06-06T07:15:48.7613019Z ##[endgroup]
2024-06-06T07:15:59.3619476Z 2024/06/06 07:15:59 Downloading https://releases.bazel.build/7.2.0/rc3/bazel-7.2.0rc3-windows-x86_64.exe...
2024-06-06T07:16:00.9729834Z ##[error]Process completed with exit code 1.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

No response

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[fmeum]

@meteorcloudy

[fmeum]

last_green fails in a similar way.

[meteorcloudy]

Let me take a look now

[meteorcloudy]

Wow, turned out to be bazelbuild/bazel@759fe7d

[meteorcloudy]

Not sure why it's not caught in presubmit

[meteorcloudy]

OK, we also did a windows VM update around the same time, maybe that's the real cause.

[fmeum]

If bazelbuild/bazel@759fe7d turns out to be the culprit, I'll never consider any change safe for cherry-picking anymore :-D

[meteorcloudy]

The binary runs fine on the same VM image where it was built:

C:\Users\pcloudy\workdir>set USE_BAZEL_VERSION=7.2.0rc3

C:\Users\pcloudy\workdir>bazel
2024/06/06 09:04:06 Downloading https://releases.bazel.build/7.2.0/rc3/bazel-7.2.0rc3-windows-x86_64.exe...
Downloading: 63 MB out of 63 MB (100%)
Extracting Bazel installation...
Starting local Bazel server and connecting to it...
                                                        [bazel release 7.2.0rc3]
Usage: bazel <command> <options> ...

Available commands:

[meteorcloudy]

I can confirm building with MSVC 2022 14.40.33807 causes this problem, but not when I switch to MSVC 2019.

[meteorcloudy]

I still haven't figured out a fix, but I can revert the Windows VM image first.

However, this would revert #1962
/cc @ahumesky @tomrenn

[meteorcloudy]

I've rebuilt Bazel binaries at HEAD, and hopefully that solves the issue.

@keertk We need to rerun the release pipeline for rc3, too.

[meteorcloudy]

OK, rc3 release have been refreshed, also fixed last_green. I'll trigger builds to fix commits on master and release-7.2.0 retrospectively.

[fmeum]

Could be related to actions/runner-images#10004 (comment)

[benjaminp]

Could 8.0.0-pre.20240530.1 be refreshed, too? It seems to have been afflicted.

[meteorcloudy]

@benjaminp 8.0.0-pre.20240530.1 should be fixed now, can you verify?

[benjaminp]

https://releases.bazel.build/8.0.0/rolling/8.0.0-pre.20240530.1/bazel-8.0.0-pre.20240530.1-windows-x86_64.exe still seems affected. (The download server reports "June 6" as the last modified time.)

[meteorcloudy]

I see, I only refreshed 3e2aec2fdefea213cde0c0fc41eb40221995d1ca. Now reran the release pipeline, should be fixed now.