Simplify rules_nixpkgs_core module extensions #423

aherrmann · 2023-09-29T12:23:36Z

Part of #183.

The previous implementation of the rules_nixpkgs_core module extensions (#344, #363, #369, #364) used a hub-repository to manage Nix repository and package import scope.

Since then, Bazel (>= 6.3) has introduced isolated module extension instances and automatic generation of use_repo stanzas. Both these features solve the same problems that the hub-repository implementation addressed, see below.

This PR replaces the previous hub-repository based implementation by a simpler implementation based on these new Bazel features. A clear advantage is that this means that users can use plain Bazel labels to refer to imported repositories and packages instead of having to access them through hub-repository accessors. This should also be an advantage in terms of WORKSPACE to bzlmod migration effort.

How isolated extensions and automatically generated use_repo address rules_nixpkgs's requirements:

Separating globally unified and locally specialized packages. New: Globally unified behavior is the default, and locally specialized behavior is achieved using an isoalted module extension instance.
Exposing imports under human friendly names. New: Because isolated module extension instances have their own namespace, we no longer need to worry about name mangling ourselves. Instead, we can generate external repositories with human friendly names directly, either the canonical attribute name in case of globally unified packages, or a human defined name in case of isolated instances or overrides in root or rules_nixpkgs_core. Thanks to the automatically generated use_repo stanzas, it is very easy for a user to manage the imports.

This uses the new isolated module extensions feature [1], instead of a hub-repository approach [2] [3], to distinguish globally unified repositories and locally specialized repositories. [1]: bazelbuild/bazel#18529 [2]: bazelbuild/bazel#17048 [3]: bazelbuild/bazel#17493

Otherwise the tests only cover the cases of rules_nixpkgs_core or the root module.

Except those that require further simple repository tags to be implemented first.

After removing the old extensions we can now use the shorter names "repository" and "package".

This is so that modules can define overrides as dev-depencies.

This is to test logic that differs between an intermediate module or the root module or rules_nixpkgs_core.

Version 2.7, 2.8, and 2.9 were removed.

aherrmann · 2023-10-02T16:41:18Z

The CI failure seems to be caused by NixOS/nixpkgs#258607. I've opened NixOS/nixpkgs#258608 with a fix. And pushed a workaround until the fix is available in nixpkgs.

The corresponding files no longer exist and the import was no longer in effect since the @nixpkgs import in docs_dependencies_2 took precedence.

To prevent the following build error on MacOS x86_64 ``` > error: don't yet have a `targetPackages.darwin.LibsystemCross for x86_64-apple-darwin` ERROR: /Users/runner/work/rules_nixpkgs/rules_nixpkgs/WORKSPACE:45:16: fetching _nixpkgs_package rule //external:coreutils_static: Traceback (most recent call last): File "/private/var/tmp/_bazel_runner/a4a3182e50c9460fa52b8c6c1e6f916f/external/rules_nixpkgs_core/nixpkgs.bzl", line 589, column 31, in _nixpkgs_package_impl _nixpkgs_build_and_symlink(repository_ctx, [nix_build_path], expr_args, build_file_content) File "/private/var/tmp/_bazel_runner/a4a3182e50c9460fa52b8c6c1e6f916f/external/rules_nixpkgs_core/nixpkgs.bzl", line 447, column 34, in _nixpkgs_build_and_symlink exec_result = execute_or_fail( File "/private/var/tmp/_bazel_runner/a4a3182e50c9460fa52b8c6c1e6f916f/external/rules_nixpkgs_core/util.bzl", line 97, column 13, in execute_or_fail fail(""" Error in fail: Cannot build Nix derivation for package '@coreutils_static'. Command: "/nix/store/x5wfa3afc4l2rc7zx3wiiwcm5j3j4lbc-nix-2.17.0/bin/nix-build" "-I" "nixpkgs=/private/var/tmp/_bazel_runner/a4a3182e50c9460fa52b8c6c1e6f916f/external/nixpkgs/nixpkgs" "-E" "import <nixpkgs> { config = {}; overlays = []; }" "-A" "pkgsStatic.coreutils" "--out-link" "bazel-support/nix-out-link" ``` https://github.com/tweag/rules_nixpkgs/actions/runs/6390957120/job/17345262106?pr=423#step:6:846 Error message origin in nixpkgs: https://github.com/aherrmann/nixpkgs/blob/59998678277cf465e749a144cf707021285bdf82/pkgs/top-level/all-packages.nix#L21799-L21802

aherrmann · 2023-10-03T12:23:35Z

The second CI failure seems to be caused by an incompatibility with more recent nixpkgs revisions and MacOS 11 on x86_64. I have not been able to track down the exact reason, but worked around the issue by falling back to pkgs.coreutils. I've opened #424 to track this.

aherrmann · 2023-10-03T13:37:13Z

CI is green now, so this is ready for review.

layus

Interesting changeset. Overall not much changed in the interface, except for some simplifications, so this is a much welcome change.

I would love to see the implications for more complex use cases of rules_nixpkgs, where @nixpkgs is not a vanilla upstream nixpkgs, but I think there is kind of a global assumption that @nixpkgs is some version of upstream nixpkgs.

I am still annoyed that bzlmod (at least for rules_nixpkgs) relies on so many conventions and unchecked assumptions, but there is not much we can do about that.

Some comments I left are merely notes for me to check after the review, these are marked with XXX.

WORKSPACE

core/MODULE.bazel

layus · 2023-10-13T15:03:21Z

flake.nix

@@ -1,6 +1,6 @@
 {
  inputs = {
-    nixpkgs.url = "github:NixOS/nixpkgs/nixos-22.11";
+    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";


Any specific reason for this change ?

Yes, to get access to Bazel version 6.3.2, which is not yet available in NixOS-22.11.

testing/core/tests/nixpkgs_repositories.bzl

layus · 2023-10-13T15:41:18Z

tests/BUILD.bazel

@@ -8,7 +8,7 @@ sh_test(
        "//nixpkgs:srcs",
        "//tests/invalid_nixpkgs_package:srcs",
        "@coreutils_static//:bin",
-        "@nix_2_7//:bin",
+        "@nix_2_10//:bin",


Any reason for thins change, except general maintenance ?

Yes, version 2.7 (and 2.8 and 2.9) are no longer available in the more recent nixpkgs revision (unstable). So we need to bump the version.

Co-authored-by: Guillaume Maudoux <[email protected]>

aherrmann added 26 commits September 27, 2023 14:54

Update to NixOS-unstable and Bazel 6.3.2

d70da96

Enable isolated extensions

c1a32e6

apply bzlmod configuration to common

e678776

Introduce an intermediate module for testing

5891452

Otherwise the tests only cover the cases of rules_nixpkgs_core or the root module.

Add simple nix_pkg extension

bbb628f

Implement attribute path overriding

ebfef9c

Add simple file and expr package tags

5dfbb3c

nixpkgs-simple --> nixpkgs

f3e541e

Move package tests to simple extension

06fce89

Except those that require further simple repository tags to be implemented first.

Add file and expr tags on the simple repo ext

73870f9

Move the remaining tests to simple extension

4742fc1

Remove the empty nix_pkg test set

277d9be

fix ref to previous loop var

12a83a2

Remove unused non-simple extensions

8254285

Remove unused nix_repo|pkg workspace shims

2890c06

Remove the "non-simple" extensions

a7343f4

Remove the module_registry support module

f3f83f6

Remove the "_simple" suffix

712bb96

After removing the old extensions we can now use the shorter names "repository" and "package".

Document the module and tag loops

e9ee233

Allow duplicates between default and dev-deps

8acd51c

This is so that modules can define overrides as dev-depencies.

Run tests in and from intermediate module

d9e0b2d

This is to test logic that differs between an intermediate module or the root module or rules_nixpkgs_core.

Test isolated extensions in intermediate module

d01acba

Fix bazelignore

332fbb0

Update the design document

b32d2c1

Allow insecure package Python2

24bbef8

aherrmann requested a review from benradf as a code owner September 29, 2023 12:23

aherrmann added 3 commits September 29, 2023 16:38

Bump fixed Nix version

c630600

Version 2.7, 2.8, and 2.9 were removed.

Patch cc-wrapper to replace --tmpdir flag

925f8e4

Add a comment to cc-wrapper patch

6222254

aherrmann added 3 commits October 3, 2023 11:08

Share patched cc between go-bzlmod and go-workspace

675f059

Remove redundant @nixpkgs

6cb237b

The corresponding files no longer exist and the import was no longer in effect since the @nixpkgs import in docs_dependencies_2 took precedence.

aherrmann force-pushed the bzlmod-isolate branch from fa796bf to 651fe0d Compare October 3, 2023 11:49

aherrmann mentioned this pull request Oct 3, 2023

pkgsStatic.coreutils is no longer available on MacOS with recent nixpkgs #424

Open

add a comment to the MacOS coreutils workaround

488358c

simplify the may_override check

72b4b1d

layus approved these changes Oct 13, 2023

View reviewed changes

Update WORKSPACE

2cb592c

Co-authored-by: Guillaume Maudoux <[email protected]>

aherrmann added the merge-queue merge on green CI label Oct 16, 2023

Merge branch 'master' into bzlmod-isolate

a86be3c

mergify bot merged commit 0fd4a8f into master Oct 16, 2023
15 checks passed

mergify bot deleted the bzlmod-isolate branch October 16, 2023 12:58

mergify bot removed the merge-queue merge on green CI label Oct 16, 2023

aherrmann mentioned this pull request Nov 1, 2023

go_tool_binary fails under remote execution (mktemp: command not found) #439

Closed

aherrmann mentioned this pull request Feb 23, 2024

Figure out a plan to stabilize isolated extension usages bazelbuild/bazel#20186

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify rules_nixpkgs_core module extensions #423

Simplify rules_nixpkgs_core module extensions #423

aherrmann commented Sep 29, 2023 •

edited

Loading

aherrmann commented Oct 2, 2023

aherrmann commented Oct 3, 2023

aherrmann commented Oct 3, 2023

layus left a comment

layus Oct 13, 2023

aherrmann Oct 16, 2023

layus Oct 13, 2023

aherrmann Oct 16, 2023

Simplify rules_nixpkgs_core module extensions #423

Simplify rules_nixpkgs_core module extensions #423

Conversation

aherrmann commented Sep 29, 2023 • edited Loading

aherrmann commented Oct 2, 2023

aherrmann commented Oct 3, 2023

aherrmann commented Oct 3, 2023

layus left a comment

Choose a reason for hiding this comment

layus Oct 13, 2023

Choose a reason for hiding this comment

aherrmann Oct 16, 2023

Choose a reason for hiding this comment

layus Oct 13, 2023

Choose a reason for hiding this comment

aherrmann Oct 16, 2023

Choose a reason for hiding this comment

aherrmann commented Sep 29, 2023 •

edited

Loading