Tracking issue for "Remote Builds without the Bytes"

[buchgr]

This issue tracks the progress towards Remote Builds without the Bytes.

Milestones

• Implement basic support for not downloading intermediate artifacts (In Bazel 0.25)
  The feature can be enabled by adding the flag --experimental_remote_download_outputs=minimal to the remote caching / execution invocation. The feature mostly works, but comes with the limitations stated in the subsequent tasks.
• Fix local actions that need to fetch remote outputs (In Bazel 0.25)
  While Bazel can run most actions remotely there are a few dozen or so action implementations that currently can't. This can be problematic because such actions often require Bazel to download remote files which this change tries to avoid. An example of such an action is the SymlinkAction.
• Implement fetching of top level artifacts (In Bazel 0.26)
  Implement --experimental_remote_download_outputs=toplevel as described in the design document.
• Support downloading unzipped undeclared test outputs
  See --remote_download_toplevel --nozip_undeclared_test_outputs does not download any undeclared test outputs for passing tests #17884 for details
• Store per output file metadata in the persistent action cache
  The persistent action cache stores the state of Bazel's output base. It does not know about remotely stored files. We need to extend it to do so. See Store remote metadata in the on disk action cache #8248
• Builds without the Bytes fails on missing AC result
  See Builds without the Bytes fails on missing AC result #10880 for details
• Support remote cache eviction
  As detailed in the design document it's possible for Bazel to wrongly think that an output file exists on the remote system because it has been evicted ("garbage collected") in the meantime. If a Bazel action then depends on an evicted file there is currently no other way but to fail the build/test. See Remote Builds without the Bytes should support remote output eviction #8250
• Add up- and download stats to the profiler
  A user should have insight into what is being up/downloaded and why.
• Be compatible with the Build Event Service local file uploader (In Bazel 0.28.0)
  This change currently doesn't interact well with the BEP and BES. See Remote Builds without the Bytes should work with BES #8249
• Don't create a runfiles tree for remote actions
  For a test that runs remotely we currently create a runfiles tree in the output base with dangling symlinks that's unused. We need to fix the SymlinkTreeStrategy.
• Download targets of symlinks in runfiles
  See --remote_download_toplevel does not download symlink targets of runfiles #18249 for details
• Support symlinks into tree artifacts
  See Action input symlink into directory doesn't work with --remote_download_minimal #15678 for details
• ctx.actions.symlink actions are not constant time
  See FR?: ctx.actions.symlink action time scales w/ size of input; should be constant #14125 for details
• Support unresolved symlinks in action outputs
  See Symlinks in action outputs are not yet supported by --experimental_remote_download_outputs=minimal #13355 for details
• Do not create runfile trees unnecessarily when building without the bytes
  See Do not create runfile trees unnecessarily when building without the bytes #18580 for details
• Bazel query silently thrashes the analysis cache when --remote_download_minimal is enabled
  See Bazel query silently thrashes the analysis cache #10902 for details
• Build event file creation fails with symlinks when using --remote_download_minimal
  See Build event file creation fails with symlinks when using --remote_download_minimal #11942 for details
• Support for mobile-install command
  See mobile-install fails with BWOB #14523 for details
• Work seamlessly with IDEs

Related Issues

#1922 #5505

[iirina]

Please assign a priority (see the maintainers guide).

[ittaiz]

@buchgr Do I understand correctly that Fix local actions that need to fetch remote outputs is a pre-requisite for developer machines to have cache hits from CI?
Also can you elaborate a bit about Work seamlessly with IDEs? What won't work off the bat?
Alternatively will we be able to run on CI with --experimental_remote_fetch_outputs=minimal and developer machines without and still have cache hits?

[ittaiz]

@buchgr does this mean the change is only one commit? (github says This branch is 1 commit ahead, 770 commits behind bazelbuild:master.)
I'd like to try it out and since we're using 0.22.0 I want to rebase your changes (if easy and applicable) on top of 0.22.0

[ola-rozenfeld]

Edit: sorry, disregard, my bad, it does compile with 0.22.0!

Which bazel version should we use to compile this with? For me, bazel build //src:bazel on this repository didn't work with any bazel releases in {0.16.1, 0.17.1, 0.19.2, 0.20.0, 0.21.0, 0.22.0}. I tried bootstrapping it, but that didn't work either. Thank you!

[moritzkiefer-da]

Hey, I was trying this branch on a very simple project today and it seems to fail to fetch binaries in some cases. I have a very simple BUILD.bazel file

cc_binary(
    name = "cbin",
    srcs = ["main.c"],
)

where main.c is just a hello world program.
I’m working in clean repository against a fresh cache started with docker run -v /tmp/bazel_empty_cache:/data -p 9090:8080 buchgr/bazel-remote-cache.

I’m now running the following commands:

~/code/bazel/bazel-bin/src/bazel run //:cbin --experimental_remote_fetch_outputs=minimal
~/code/bazel/bazel-bin/src/bazel clean --expunge
~/code/bazel/bazel-bin/src/bazel run //:cbin --experimental_remote_fetch_outputs=minimal

The first invocation works completely fine and outputs "Hello World". However, after cleaning the cache this fails and I get the following output:

Starting local Bazel server and connecting to it...
INFO: Invocation ID: c2199026-669c-4271-9412-fb8abce40f1b
INFO: Analysed target //:cbin (8 packages loaded, 58 targets configured).
INFO: Found 1 target...
Target //:cbin up-to-date:
  bazel-bin/cbin
INFO: Elapsed time: 3.058s, Critical Path: 0.56s
INFO: 2 processes: 2 remote cache hit.
INFO: Build completed successfully, 6 total actions
ERROR: Non-existent or non-executable /home/moritz/.cache/bazel/_bazel_moritz/9461a41f957a379ce0aaa8755f8e70f9/execroot/__main__/bazel-ouINFO: Build completed successfully, 6 total actions

So the binary was never downloaded. The problem appears to be in RemoteSpawnCache, if I change the MINIMAL case to behave like ALL it is downloaded again but ofc that is not the solution as it goes against the whole purpose of this patch. I am not entirely sure how this is supposed to work. If I change CppLinkAction to add the output to requiredLocalOutputs then it also works but if I understand the patch correctly this shouldn’t be necessary

[meisterT]

The commits should be part of 7.0.0-pre.20230628.2 that we released yesterday.

[purkhusid]

I encountered issue with the latest pre release where I had --remote_download_minimal enabled and Bazel failed to find files when doing bazel run. Are there still known issues with that use case?

[tjgq]

@purkhusid Not that I'm aware of - do you have a repro?

[purkhusid]

I don't have a minimal repro, this is part of a larger repo. Things worked again after I added:

run --remote_download_outputs=toplevel
run --build_runfile_links

to our bazelrc

If it works with toplevel, it should also work with minimal, right? I'll see if I can narrow this down a bit and extract a minimal repro.

[purkhusid]

@tjgq I've looked a bit into this and my build breaks as soon as I add --nobuild_runfile_links. This was not the case in 6.3.2.

I'll see if I can narrow it down a bit more and see in what build it started to fail.

[tjgq]

@purkhusid I find that surprising, because until 24ba4fa (which is not included in the latest pre-release, 7.0.0-pre.20230823.4), --remote_download_minimal implied --nobuild_runfile_links, so setting the latter manually should have been a no-op. But maybe you're using--remote_download_outputs=minimal (which has never implied --nobuild_runfile_links) or maybe you're running a Bazel that includes 24ba4fa (so that --nobuild_runfile_links is no longer implied by --remote_download_minimal). I'm sorry that this is so confusing...

Also - are you by any chance running on Windows? (Asking because this might be somehow related to #19333.)

Anyway, I've just tried bazel running a trivial executable rule with runfiles, and it seems to be able to locate them at runtime, even with --remote_download_minimal, both before and after 24ba4fa. So a reproducer (+ the OS and the exact Bazel version you're using) would be welcome.

[purkhusid]

Just to be clear I explicitly set:

--remote_download_outputs=minimal
--nobuild_runfile_links

And am not using --remote_download_minimal

This worked in 6.3.2 but not in the latest prerelease. If I remove --nobuild_runfile_links everything works again. Is it possible to accidentally design a rule so that it does not work with bazel run and --nobuild_runfile_links?

[tjgq]

If the file you're trying to access is indeed included in the runfiles for the target you're trying to bazel run (as returned in the rule's DefaultInfo) but it cannot be found at its expected location at run time, then I'd consider that a bug, irrespective of what --remote_download_outputs or --nobuild_runfile_links are set to.

[purkhusid]

Alrighty, I managed to narrow this down a bit. You actually have to run bazel build on the target before you run bazel run.

If I run bazel clean and then bazel run on the target then everything works fine.
If I run bazel clean and then bazel build and then bazel run then it fails to find the files.

[purkhusid]

Another update. Maybe I should have looked into it in the beginning but if I use last_green it seems like the issue is no longer present.

[purkhusid]

Ok, this seems to be an issue in the latest pre release and on last_green. The step I have to reproduce it is still:

If I run bazel clean and then bazel run on the target then everything works fine.
If I run bazel clean and then bazel build and then bazel run then it fails to find the files.

This is with --remote_download_outputs=minimal and --nobuild_runfile_links

[bjacklyn]

When I tried this a few years ago I ran into issues where bazel would crash if the remote cache said it had a file but didn't (which could happen due to cache eviction, original file upload failed, or any other reason).

I'm wondering if these issues have been resolved -- suppose I created a remote cache called "lying cache" which randomly lies and says it has AC/CAS files that it doesn't. Will bazel gracefully handle this and perform "action rewinding" in all cases?

[tjgq]

@bjacklyn See #8250 (comment).

[tjgq]

@purkhusid Do you mind giving last_green a try to see if it fixes the issue you were describing above? The recently submitted 84d1a72 improved the incremental correctness of --nobuild_runfile_links.

[purkhusid]

@tjgq The problem still exists on last_green. I'm trying to extract a minimal reproduction.

[purkhusid]

@tjgq I have managed to do a pretty minimal reproduction: https://github.com/purkhusid/bazel_runfiles_repro
You can look at https://github.com/purkhusid/bazel_runfiles_repro/blob/main/reproduce.sh to see what commands work and don't work.

I managed to narrow it down to the interaction between these two flags:

--noexperimental_check_output_files
--remote_download_outputs=minimal

I've got --noexperimental_check_output_files set in our .bazelrc as an performance improvement but It's perfectly acceptable that I remove this from our bazelrc if this is combination of flags is expected to work in some strange way.

[tjgq]

@purkhusid Thanks for the repro. My initial reaction is that it's a bug (I don't see why those two flags would be incompatible) but we're going to need some more time to figure out where the issue is.

[tjgq]

Let's move the discussion to the separately filed #20843.