Add --experimental_aot_cache_training_run to speed up server startup with a JDK AOT cache

[fmeum]

Description

The new --experimental_aot_cache_training_run startup option starts a new server that records an ahead-of-time cache of the classes it loads and links as well as method profiles (JEP 483, JEP 514, JEP 515) via -XX:AOTCacheOutput. Later invocations without the option keep using that server, so the training run consists of all commands run until the server exits, typically via bazel shutdown:

bazel --experimental_aot_cache_training_run build //...
bazel test //...
bazel shutdown

The JVM then assembles the cache as <install_base>.aot next to the install base and every server started for the same installation of Bazel afterwards uses it via -XX:AOTCache, regardless of the output base.

Measured with a build of this PR and its embedded JDK 26 (AOT class linking disabled, see below) on an Apple M3 Max after a training run consisting of build, test, query, cquery, aquery and info on the examples/ targets, cold commands against a fresh server are 25-40% faster (medians of 5 runs, warm disk cache):

Command against a fresh server	No cache	AOT cache
bazel version	1.04s	0.81s
bazel info	1.46s	1.04s
bazel query 'deps(//examples/java-native/...)'	1.58s	1.18s
bazel build //examples/cpp/... //examples/java-native/... (fully cached)	3.22s	2.57s
bazel test //examples/shell/... //examples/gen/... (fully cached)	2.95s	2.36s

Warm commands are unaffected.

Details:

• All AOT cache JVM arguments are volatile startup options: a cache appearing next to the install base doesn't restart a running server and neither does the absence of the training run option. An invocation with the option always restarts the server instead, since the JVM only records from the moment it starts.
• The JVM writes the cache header last, so an interrupted dump leaves it zeroed. The client only passes complete caches to the JVM, which would otherwise reject the cache on every start.
• The JVM treats some kinds of unusable caches as fatal errors during startup rather than ignoring them. If the server crashes during startup while using the cache, the client deletes it and writes a .disabled marker that prevents the use of a cache for this install base until a new training run records one.
• AOT class linking is disabled for now since JDK 26 fails to start with such a cache if the JDK is a jlinked image such as the embedded one (JDK-8381222, which doesn't affect JDK 25 or 27). The cache still contains the parsed and verified classes and method profiles, which provides most of the benefit.
• The cache is neither used nor recorded with --host_jvm_debug, since the JVM refuses to load one with a JDWP agent attached (JDK-8349122) and a debugging session isn't a representative training run.
• The install base garbage collector deletes the cache and its marker files together with a stale install base.

Motivation

Server startup is a noticeable part of the latency of short Bazel invocations, in particular in CI and for tools that frequently start fresh servers. The JDK's AOT cache skips most of the class loading, verification and linking work and starts out with method profiles, but it requires a training run that is representative of the actual workload, which only the user can provide.

Build API Changes

No

Release Notes

RELNOTES[NEW]: The new --experimental_aot_cache_training_run startup option starts a server that records a JDK AOT cache when it is shut down. All servers started afterwards for the same installation of Bazel use the cache to start up faster.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.