CacheNotFoundException when resuming build after remote cache evicted objects #19348
Comments
iancha1992
added
the
team-Remote-Exec
oquenchil
added
P2
|
I agree the correct expectation is for Bazel to rerun the action. You mentioned you had several retries, did they fail with the same missing digests? |
|
Yup they failed with the same missing digests. |
Facing this issue again today on Bazel 6.4.0rc1. Issue goes away on immediate retry 🤔 |
|
Was there any automatic retries? |
|
No auto retry for me. |
|
It looks like in this corner case Bazel wasn't able to detect the cache eviction error and retry. NOT Bazel wasn't able to clear the stale state.
|
|
I'm running into the same problem with |
|
cc: @coeuvre |
|
This should have been fixed by eda0fe4. If not, please reopen. |
|
looks like issue still exists in 7.3.1 version. |
|
There is another underlying issue which should be fixed by 9187a7e and is included in the upcoming 7.4.0 release. |
|
tried the version "8.0.0rc1" looks like still have the issue. GoCompilePkg external/gazelle++go_deps+org_golang_x_sys/unix/unix.a; 0s remote, remote-cache ... (4 actions running) |
|
The exception is thrown by https://cs.opensource.google/bazel/bazel/+/master:src/main/java/com/google/devtools/build/lib/remote/RemoteExecutionCache.java;l=199;drc=51bb7b820b80b437c2fc229f49904d3c57df2b4b which means the blob was probably evicted by the remote cache during the build. How is your remote execution server setup? Did |
|
the remote execution setup is the bazelbuildfarm. the --experimental_remote_cache_eviction_retries looks like can help in such case, but it still print many many Exceptions log which is a bit annoying. |
|
This this case, consider giving your remote cache a larger storage space so it wouldn't evict blobs that often. |
Description of the bug:
Our remote cache evicts old objects in an LRU fashion.
This means that if you have not built for a while, especially if your config is unique, it's likely that your cache object will get evicted.
Today I resumed my M1 Macbook laptop after a weekend break and got a stack trace like this after my first build
This is not fixed after several retries, but it seems to be fixed after I went for lunch and came back to the laptop (no changes made). Assuming this was caused by of idle shutdown of Bazel JVM.
I think the correct expectation here is for Bazel to tell the remote cache / remote executor to re-run the action, but it seems like there could be edge cases that are not being handled properly.
Which category does this issue belong to?
Remote Execution
What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.
Not sure just yet.
We have "build without bytes" turned on with GRPC cache (no disk cache) and remote execution enabled.
Which operating system are you running Bazel on?
MacOS 13.5.1 darwin64
What is the output of
bazel info release?release 6.3.1
If
bazel info releasereturnsdevelopment versionor(@non-git), tell us how you built Bazel.No response
What's the output of
git remote get-url origin; git rev-parse master; git rev-parse HEAD?No response
Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.
Irrelevant as the issue would go away if the JVM restart.
Have you found anything relevant by searching the web?
No response
Any other information, logs, or outputs that you want to share?
The problem is not exclusive to MacOS environment, however.
We have seen similar reports from our customers, who set up their Linux CI against our remote cache as well.
Typically, it would be something like: do first build, wait several weeks, remote cache evicted, do second build -> similar failure.
Because of this, we have not been able to reproduce the situation reliably.
The text was updated successfully, but these errors were encountered: