Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] FileNotFoundError jaraco/text/Lorem ipsum.txt when importing pkg_resources #4487

Closed
shk3 opened this issue Jul 18, 2024 · 7 comments
Closed

Comments

@shk3
Copy link

shk3 commented Jul 18, 2024

setuptools version

71.0.0

Python version

Python 3.9

OS

Ubuntu Linux & macOS

Additional environment information

No response

Description

This seems to be a regression of #3072. My suspicion is commit d03cd0e might play a part of it, but I don't know enough about Python's package management to be sure.

We are building with an older version of Sphinx with Bazel, which effectively import pkg_resources just like what has happened in #3072, which is supposed to have been fixed via #3077.

Expected behavior

Pinning setuptools==70.3.0 in requirements.txt works fine. This issue started with setuptools==71.0.0 as well as setuptools==71.0.1.

How to Reproduce

Pretty much the same as #3072, except in our case we were building an old Sphinx 4.1.0, which doesn't import pkg_resources anymore in new versions. We are reporting because this could happen to other projects that import pkg_resources in a later version.

Using Bazel and rules_python, build a zip executable using:

bazel build //<target> --output_groups=python_zip_file
python3 bazel-bin/<target>/<target>.zip 

Output

Traceback (most recent call last):
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/repo/<file>.py", line 8, in <module>
    from sphinx.ext.apidoc import main as sphinx_api_doc
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/ext/apidoc.py", line 30, in <module>
    from sphinx.cmd.quickstart import EXTENSIONS
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/cmd/quickstart.py", line 40, in <module>
    from sphinx.util.template import SphinxRenderer
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/util/template.py", line 22, in <module>
    from sphinx.jinja2glue import SphinxFileSystemLoader
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/jinja2glue.py", line 20, in <module>
    from sphinx.application import TemplateBridge
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/application.py", line 43, in <module>
    from sphinx.registry import SphinxComponentRegistry
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__sphinx/site-packages/sphinx/registry.py", line 24, in <module>
    from pkg_resources import iter_entry_points
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__setuptools/site-packages/pkg_resources/__init__.py", line 93, in <module>
    from jaraco.text import (
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__setuptools/site-packages/setuptools/_vendor/jaraco/text/__init__.py", line 231, in <module>
    files(__name__).joinpath('Lorem ipsum.txt').read_text(encoding='utf-8')
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/external/python3_9_x86_64-apple-darwin/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/external/python3_9_x86_64-apple-darwin/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/external/python3_9_x86_64-apple-darwin/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/username/.cache/bazel/6e5f220bf69d4226b76b935c3f89be91/execroot/repo/bazel-out/darwin-fastbuild/bin/<target>.runfiles/pip/pypi__setuptools/site-packages/setuptools/_vendor/jaraco/text/Lorem ipsum.txt'
@shk3 shk3 added bug Needs Triage Issues that need to be evaluated for severity and status. labels Jul 18, 2024
@osfrickler
Copy link

I think this should be fixed by the just released 71.0.2, can you check against that?

@allenbenz-doma
Copy link

Seeing the same issue, wasn't fixed for me in 71.0.3.

Writing down more context to save someone else time.

bazel doesn't support whitespaces in paths There's an experimental flag that reportedly can be used if you're using bazel 7.0.0 or above.

The previous workaround looks like it was removed in this PR
Specifically this commit and this other commit removed the a regex filter that was deleting the bit of code loading the Lorem ipsum.txt file in the vendored copy of jaraco.

jaraco is unlikely to rename the file.

@jaraco
Copy link
Member

jaraco commented Jul 18, 2024

Thanks for the report and detailed analysis. And sorry for the inconvenience while Setuptools works to clean up some cruft that's accumulated.

I'd forgotten that I'd built in a patch for jaraco.text when vendorizing it. I was really happy to be rid of all that tech debt in 9234fc3, but I'd been working from the assumption that the patches were there solely to support vendorizing and not there to support other issues, so the removal of the workaround introduced in #3077 was unintentional.

As setuptools moves toward a regime that doesn't depend on vendored packages, any workarounds that apply to the vendored package will become less viable. Setuptools 71 has a preference for naturally-installed packages. What happens in a Bazel-managed environment if jaraco.text is installed naturally?

bazel doesn't support whitespaces in paths There's an experimental flag that reportedly can be used if you're using bazel 7.0.0 or above.

That's great news! I'd really like to know (a) if employing that flag fully addresses the issue, and (b) how viable it would be for users to adopt Bazel 7.

Assuming that's not a path to a long-term solution, is there another path by which setuptools and/or jaraco.text can be patched on import to the Bazel environment? I'd like for Setuptools not to have to bear the responsibility for compatibility with a downstream tool. I know at Google, which uses a similar tool, the jaraco.text was imported with a patch that renames the file to make it viable in that environment. Is there a way to do something similar for //target?

@shk3
Copy link
Author

shk3 commented Jul 19, 2024

I can confirm that this issue still exists in 71.0.2 and 71.0.3. We are running Bazel 6.4.0. I'm not sure about what the effort would be for upgrading, so I'm leaving it for other folks who might be impacted by this to tell.

As for my particular use case, we were able to fix by upgrading the Sphinx version, which does not import pkg_resources anymore. So, I'm no longer impacted.

@w4-donguk
Copy link

I can confirm that this issue still exists in 71.0.2 and 71.0.3. We are running Bazel 6.4.0. I'm not sure about what the effort would be for upgrading, so I'm leaving it for other folks who might be impacted by this to tell.

As for my particular use case, we were able to fix by upgrading the Sphinx version, which does not import pkg_resources anymore. So, I'm no longer impacted.

which version did you updated to?

@shk3
Copy link
Author

shk3 commented Jul 23, 2024

@w4-donguk We upgraded Sphinx from 4.1.0 to 7.4.5. They removed the import of pkg_resources since 4.4.0.

FWIW, the Sphinx upgrade only resolved our issue by not importing pkg_resources. For other modules that really need to import pkg_resources, they would still encounter this issue with the latest setuptools with Bazel.

I'm reporting mainly to make sure the community is aware of this unexpected side-effect. I agree with @jaraco that setuptools probably shouldn't really compromise for all kind of weirdness in the downstream systems. I personally haven't really got the time to dive into Bazel to see if there could be patches on Bazel rules to get it work as folks suggested.

@jaraco
Copy link
Member

jaraco commented Jul 30, 2024

I'd prefer to do nothing here and to use the package naturally, and rely on downstream tooling to account for patching any jaraco.text or to employ the proper fix using the latest Bazel, especially because the latest Setuptools no longer has control over which jaraco.text is used (as it will prefer a site-installed copy to its own).

Actually, that gives me another idea for a workaround - if bazel (or the tooling around it) had a way to install a patched version of jaraco.text (either without that property or with it renamed), Setuptools would use that version and not its bundled copy. One could package that up as jaraco.text-bazel-safe and install it when installing Setuptools. It's ugly, but at least it places the maintenance responsibility nearer the cause.

I'm going to close this for now, but feel free to follow-up, especially if the proposed workarounds turn out to be too onerous, and we might consider re-introducing the patch of the vendored copy.

@jaraco jaraco closed this as completed Jul 30, 2024
@jaraco jaraco added wontfix downstream technical debt and removed Needs Triage Issues that need to be evaluated for severity and status. labels Jul 30, 2024
mum4k pushed a commit to envoyproxy/nighthawk that referenced this issue Aug 29, 2024
Update Nighthawk Python dependencies, pinning setuptools<=70.3.0.

Due to Bazel not supporting spaces in filenames, we have to pin setuptools to a version earlier than 71.0.0, which moves away from vendorized packages towards naturally installed ones. This eliminated a workaround which unintentionally allowed  Bazel to use the jaraco.text package despite it using a filename with a space in it (which usually doesn't play nice with Bazel): pypa/setuptools#4487 (comment).

We can remove this pin when Envoy (and therefore Nighthawk) is migrated to build with Bazel 7. This will allow us to build with an experimental flag that allows spaces in filenames: https://bazel.build/reference/command-line-reference#flag--experimental_inprocess_symlink_creation.

Signed-off-by: Sebastian Avila <[email protected]>
CheckmkCI pushed a commit to Checkmk/checkmk that referenced this issue Oct 1, 2024
See pypa/setuptools#4487 (comment)
for context as to why we're pinning.

CMK-19117

Change-Id: Ib06f2a4faa0ee0eb1ed103b72850218ee27a3dc0
copybara-service bot pushed a commit to google-ai-edge/mediapipe that referenced this issue Oct 8, 2024
Fix setuptools=70.3.0 due to pypa/setuptools#4487

PiperOrigin-RevId: 683648416
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants