Namespace package __path__ manipulation breaks packages using __init__.py to define code

[dzbarsky]

🐞 bug report

Affected Rule

https://github.com/bazelbuild/rules_python/blob/fe33a4582c37499f3caeb49a07a78fc7948a8949/python/pip_install/tools/wheel_installer/namespace_pkgs.py#L90-L99

Is this a regression?

Not sure, first time I have tried this.

Description

The code in the package's source init.py seems to get overwritten. I'm not fully sure why, it appears that https://github.com/bazelbuild/rules_python/blob/fe33a4582c37499f3caeb49a07a78fc7948a8949/python/pip_install/tools/wheel_installer/namespace_pkgs.py#L87-L88 is trying to guard against it.

If possible, it would be nice to lay out a site_packages that is less hacky.

🔬 Minimal Reproduction

https://github.com/dzbarsky/rules_python_pylink_bug

bazel test //:flink_import_test

🔥 Exception or Error

ImportError: cannot import name 'add_version_doc' from 'pyflink' (/private/var/tmp/_bazel_zbarsky/45e7e567ddccd93ad1b41c75488b912d/sandbox/darwin-sandbox/13/execroot/__main__/bazel-out/darwin-fastbuild/bin/flink_import_test.runfiles/pip_apache_flink_libraries/site-packages/pyflink/__init__.py)

🌍 Your Environment

Operating System:

Output of bazel version:

  
Bazelisk version: development
Build label: 6.3.2
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Aug 8 15:58:06 2023 (1691510286)
Build timestamp: 1691510286
Build timestamp as int: 1691510286
  

Rules_python version:

  
0.25
  

Anything else relevant?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity for 180 days. It will be closed if no further activity occurs in 30 days.
Collaborators can add an assignee to keep this open indefinitely. Thanks for your contributions to rules_python!

[dzbarsky]

This is still an active issues, please keep open

[groodt]

Potentially related: #2156

I'm not completely sure that this is caused by lack of site-packages, so I won't close this as a duplicate.

[engnatha]

I was able to get around this by setting enable_implicit_namespace_pkgs = True in the pip_parse target.

[engnatha]

To follow up on my comment above, while this does allow the import to work, it wipes any data directories that don't have any python files. There's some flink directories that contain only java archive files. Still discovering when precisely those are needed.

[groodt]

I've got to the bottom of the issue here. The issue here is indeed solved (or handled better) by #2156 Therefore, closing as duplicate.

Root cause:

• apache-flink-libraries and apache-flink are not correctly setup as namespace packages (they are neither pkgutil or implicit namespace packages)
• apache-flink==1.17.1 includes pyflink/__init__.py which means that it isn't an implicit namespace package. It's also not a pkgutil namespace package due to the contents. See: https://pypi-browser.org/package/apache-flink/apache_flink-1.17.1-cp39-cp39-manylinux1_x86_64.whl
• apache-flink-libraries==1.17.1 also shares a folder called pyflink, but it doesn't include a __init__.py so it is a namespace package which shadows the pyflink namespace when included on PYTHONPATH

This only works by accident outside bazel, because pip and others all just unzip the files into the same site-packages folder 😂

Suggestions here would be:

• Fix the packaging issue upstream
• Patch the sdist and/or wheel using bazel or outside bazel so that the packages don't shadow namespaces
• See Implement venv/site-packages based binaries #2156 for further options

Further references:

• PEP 420 for details on pkgutil and implicit namespace packages: https://peps.python.org/pep-0420/
• RealPython article on namespace packages https://realpython.com/python-namespace-package/
• Link to see how apache-flink==1.17.1 is not a namespace package https://pypi-browser.org/package/apache-flink/apache_flink-1.17.1-cp39-cp39-manylinux1_x86_64.whl