Gracefuly handle available CPU cores

[blackliner]

Is your feature request related to a problem? Please describe.
As described in https://nixos.org/manual/nix/stable/advanced-topics/cores-vs-jobs, and it applies to bazel too, it can happen that you trash your system, or even go into OOM issues, when too many parallel nix derivates are being built. The goal is to reduce this number to the number of total available cores (mind: cgroups (!) in containers), to avoid this. Preferable in a dynamic way.

Describe the solution you'd like
The nix build should be executed with a limited number of available cores. Not sure if this is possible, but a good solution would also make sure that a single nix build will also "consume" multiple action slots in bazel, so bazel also "knows" about the load and does not spawn too many other things in parallel.

Describe alternatives you've considered
In our CI, we hardcode NIX_BUILD_CORES=2, but this does not have an effect.

Additional context
N/A

[benradf]

Not sure if this is possible, but a good solution would also make sure that a single nix build will also "consume" multiple action slots in bazel, so bazel also "knows" about the load and does not spawn too many other things in parallel.

Agreed, this would be ideal. Not sure if you've seen it already, but there's an open Bazel issue on this topic: Cooperative Parallelism #10443.

Essentially we need Bazel to provide some kind of API that we can use to cooperatively share available cores.

[blackliner]

Thanks, yes that issue looks a) very promising and b) very stale 🤣

But bazelbuild/bazel#10443 (comment) made a good suggestion: resource_set
Can we use them to assign the nix rules one of these "resources" (lets say we have 4 of them), and then call nix with 4 cores on a 16 core machine? (TBH not 100% sure how resource_set works)

[blackliner]

for the time being, is there any way for rules_nixpkgs to honor NIX_BUILD_CORES?
This doesn't work:

NIX_BUILD_CORES=2 bazel build @some_nix_dependency//...
...
build flags: -j32 -l32

https://bazel.build/reference/command-line-reference#flag--repo_env doesn't work either ;-/

[benradf]

I think the causality is reversed — you need to set the cores config, and then Nix will ensure NIX_BUILD_CORES is set to that value when the derivation gets built (though the derivation might choose to ignore it).

You can set cores globally in ~/.config/nix/nix.conf, or via the NIX_CONFIG environment variable:

bazel build @some_nix_dependency//... --repo_env=NIX_CONFIG='cores = 2'