Using --experimental_remote_cache_lease_extension causes fast builds to wait for lease extensions to complete

[brentleyjones]

Description of the bug:

When using --experimental_remote_cache_lease_extension a fast build that would normally take 1-2 seconds can take much longer waiting for the serial background lease extensions to finish

https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/RemoteLeaseExtension.java;l=127-132;drc=3beaaaf23e4f6e9071ef7eabced7d64513573e80

One or both of two things should be different:

• Some sort of batching/parallel refreshing should happen
• Don't block the end of the build. Allow the extensions to continue to happen in the background. Similar to BES uploads when set to async.

Which category does this issue belong to?

No response

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Run a bazel build //... --experimental_remote_cache_lease_extension --experimental_remote_cache_ttl=5m. Wait for blobs to expire (5 minutes). Run it again. See that the build waits for the lease extensions to finish before completing the build.

Which operating system are you running Bazel on?

No response

What is the output of bazel info release?

release 7.2.0rc1

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse HEAD ?

No response

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[coeuvre]

I want lease extension to have less impact on the build performance so i would like to avoid batching. But blocking in the end is bad. Another option is to cancel the extension in the end of the build.

[coeuvre]

I cannot reproduce it. There is indeed a slowdown during the incremental build after the TTL is expired. But the slowdown is due to Bazel has to discard the local action cache and check for remote cache, not due to lease extension.

Can you share the json trace profile, or check whether there are checkOutputs spans in the profile?

[brentleyjones]

Hmm, yeah, in my testing it does seem about the same with and without it. I think I was getting unlucky with my previous tests. There are checkOutputs in both versions.

[brentleyjones]

Though, these are no-op builds (building the same build over and over again, all of them successful), why are there so many action dependency checking? I DMed you a log on Slack.

[coeuvre]

The root cause is #22367.