Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redact absolute root module file path in MODULE.bazel.lock #18949

Closed

Conversation

fmeum
Copy link
Collaborator

@fmeum fmeum commented Jul 15, 2023

The absolute path of the root module file in Starlark Locations is replaced with a constant when writing to the lock file.

Work towards #18936

@fmeum fmeum changed the title Relativize absolute root module file path in MODULE.bazel.lock Redact absolute root module file path in MODULE.bazel.lock Jul 15, 2023
@fmeum fmeum force-pushed the 18936-absolute-paths-in-lock-file branch 5 times, most recently from cb8928e to 1929128 Compare July 17, 2023 10:55
@fmeum fmeum marked this pull request as ready for review July 17, 2023 11:09
@github-actions github-actions bot added awaiting-review PR is awaiting review from an assigned reviewer team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. labels Jul 17, 2023
@fmeum fmeum requested a review from SalmaSamy July 17, 2023 11:09
@fmeum
Copy link
Collaborator Author

fmeum commented Jul 17, 2023

The new test is failing on Windows:

======================================================================
FAIL: testNoAbsoluteRootModuleFilePath (__main__.BazelLockfileTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\temp\Bazel.runfiles_ugd2_j1m\runfiles\io_bazel\src\test\py\bazel\bzlmod\bazel_lockfile_test.py", line 553, in testNoAbsoluteRootModuleFilePath
    self.assertIn(
AssertionError: 'ERROR: module extension "other_ext" from "//:extension.bzl" does not generate repository "hello", yet it is imported as "other_ext_hello" in the usage at C:/b/u4ube5d5/execroot/io_bazel/_tmp/b19cb386875f88e3fcf9655c858fde97\\tests_root\\tmpu1fpr48_\\MODULE.bazel:4:26' not found in ["$TEST_TMPDIR defined: output root default is 'c:\\b\\u4ube5d5\\execroot\\io_bazel\\_tmp\\b19cb386875f88e3fcf9655c858fde97' and max_idle_secs default is '15'.", 'Starting local Bazel server and connecting to it...', 'Loading:', 'Loading:', 'Loading: 0 packages loaded', 'ERROR: module extension "other_ext" from "//:extension.bzl" does not generate repository "hello", yet it is imported as "other_ext_hello" in the usage at C:/b/u4ube5d5/execroot/io_bazel/_tmp/b19cb386875f88e3fcf9655c858fde97/tests_root/tmpu1fpr48_/MODULE.bazel:4:26', 'INFO: Elapsed time: 3.178s', 'INFO: 0 processes.', 'ERROR: Build did NOT complete successfully']

This could be seen as a "real" issue: The location path doesn't use the native path separator. What do you think, should we fix that or just ignore it in the test for now?

Edit: Nvm, this appears to be the default in all Starlark stack traces. I will just adapt the test, which generates a weird mix of separators anyway.

The absolute paths of the root module file path in Starlark `Location`s
is replaced with a constant when written to the lock file and replaced
back when reading from the file.

Work towards bazelbuild#18936
@fmeum fmeum force-pushed the 18936-absolute-paths-in-lock-file branch from 1929128 to a9aa79a Compare July 17, 2023 12:49
Copy link
Member

@meteorcloudy meteorcloudy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks for the fix!

@SalmaSamy please take a final look!

@meteorcloudy meteorcloudy added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Jul 17, 2023
@SalmaSamy SalmaSamy added awaiting-review PR is awaiting review from an assigned reviewer and removed awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally labels Jul 17, 2023
protected abstract static class RootModuleFileEscapingLocation {
// This marker string is neither a valid absolute path nor a valid URL and thus cannot conflict
// with any real module file location.
private static final String ROOT_MODULE_FILE_LABEL = "@@//:MODULE.bazel";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you for the fix, but I was thinking that we could just actually use the label in error messages too (i.e. change the logic in ModuleFileFunction). IMO it's pretty clear which file @@//:MODULE.bazel (or better, if we use Label#getDisplayForm(), //:MODULE.bazel) is referring to. That would also remove this special-casing at lockfile read/write time. WDYT?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that absolute paths still provide the best usability here: You can often just copy&paste the path together with the line number to go to the correct place in your editor. All other Starlark stack traces also contain absolute paths in this place. Building //:MODULE.bazel or querying for it also results in an error about the file not being exported.

I view the switch to labels as a tradeoff that reduces implementation complexity at the cost of usability. This is definitely warranted for non-registry overrides, which are a bit more of an "advanced" concept anyway. But if we can keep absolute paths for the root module file, which I expect to cause most errors arising in practice, with a reasonable amount of complexity, then we should consider doing that.

I am actually not sure anymore whether the display form is better for non-root modules: The canonical form has the advantage that it more directly corresponds to the external path on disk, which is ultimately what the user wants to find if they face an error.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely warranted for non-registry overrides, which are a bit more of an "advanced" concept anyway.

To be clear, is your preference to use labels for non-registry overrides in error messages?

I am actually not sure anymore whether the display form is better for non-root modules: The canonical form has the advantage that it more directly corresponds to the external path on disk, which is ultimately what the user wants to find if they face an error.

I don't really mind it either way. The difference would only be @foo//:MODULE.bazel vs @@foo~override//:MODULE.bazel, which is not a huge deal IMO.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is definitely warranted for non-registry overrides, which are a bit more of an "advanced" concept anyway.

To be clear, is your preference to use labels for non-registry overrides in error messages?

Ideally we would also keep absolute paths for them. But I can see how that would introduce too much complexity (tracking and replacing the output base) for too little gain (non-registry overrides were explicitly added by the user, so they should have a better idea where their module files live). That's why I consider labels a decent tradeoff in this case.

I am actually not sure anymore whether the display form is better for non-root modules: The canonical form has the advantage that it more directly corresponds to the external path on disk, which is ultimately what the user wants to find if they face an error.

I don't really mind it either way. The difference would only be @foo//:MODULE.bazel vs @@foo~override//:MODULE.bazel, which is not a huge deal IMO.

Couldn't it be @my_gazelle//:MODULE.bazel vs @@gazelle~override//:MODULE.bazel? In that case, the former could make it harder to locate the directory in external. But of course it's also more easily recognized by the user, so I'm not completely sure which I like better.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. This is holding up 6.3.0, and while I'm not a huge fan of the special de/serialization logic, I don't have a better idea right now. So let's get this submitted (and I'll fast track the label fix for non-registry overrides). Thanks!

@Wyverald Wyverald added awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally and removed awaiting-review PR is awaiting review from an assigned reviewer labels Jul 18, 2023
@github-actions github-actions bot removed the awaiting-PR-merge PR has been approved by a reviewer and is ready to be merge internally label Jul 18, 2023
@iancha1992
Copy link
Member

@bazel-io fork 6.3.0

@iancha1992
Copy link
Member

iancha1992 commented Jul 18, 2023

@Wyverald Does this need to be cherry-picked to release-6.3.0?
If so, then how should we go about cherry-picking it because looks like there's a conflict with the "src/test/py/bazel/bzlmod/bazel_lockfile_test.py"

cc: @bazelbuild/triage

@Wyverald
Copy link
Member

yes we should cherry-pick it. I'm no longer at work but can send a PR tomorrow morning.

@fmeum fmeum deleted the 18936-absolute-paths-in-lock-file branch July 19, 2023 10:59
Wyverald added a commit that referenced this pull request Jul 19, 2023
The absolute path of the root module file in Starlark `Location`s is replaced with a constant when writing to the lock file.

Work towards #18936

Closes #18949.

PiperOrigin-RevId: 549118781
Change-Id: Ie689f7b5edf92296772c605845d694d872074214
iancha1992 pushed a commit that referenced this pull request Jul 19, 2023
* Redact absolute root module file path in `MODULE.bazel.lock`

The absolute path of the root module file in Starlark `Location`s is replaced with a constant when writing to the lock file.

Work towards #18936

Closes #18949.

PiperOrigin-RevId: 549118781
Change-Id: Ie689f7b5edf92296772c605845d694d872074214

* Use a label as the location of the MODULE.bazel file of non-registry overrides

Fixes #18936

PiperOrigin-RevId: 549310245
Change-Id: I852570fbc81c1592c2fc0b3848d4b58e1c9ffb7d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants