Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ensure no missing pip dependencies in build_image.py #218

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

ANogin
Copy link
Collaborator

@ANogin ANogin commented Feb 9, 2023

One sentence summary of this PR (This should go in the CHANGELOG!)
This changes the make install/make develop step of the finish.Dockerfile to run with network disabled, which means that any missing dependencies that were not properly installed in base.Dockerfile result in a build failure and flagged.

Link to Related Issue(s)
Would enable detecting things like #419 would be detected by CI. This depends on #571 fixing #419 (until that is fixed, this change would break the build - will keep it in draft state until then).

Please describe the changes in your request.
See summary.

Anyone you think should look at this, specifically?
@whyitfor

@ANogin ANogin mentioned this pull request Jul 27, 2023
1 task
@ANogin ANogin marked this pull request as ready for review December 20, 2023 23:34
@ANogin
Copy link
Collaborator Author

ANogin commented Feb 12, 2024

@whyitfor could we please land this one and use the new option in our CI? As I am working on #416 and #417 I am went to go back to this manually to make sure all the dependencies across ofrak components are consistent (only remembered to do it because I saw the docker build output for #417 and noticed it's uninstalling/reinstalling some packages in weird way in make develop) - and now discovering that even the main branch has incorrectly specified dependencies (will fill in separate PRs on those)?

@whyitfor
Copy link
Contributor

@ANogin, can you complete the PR descripion for this?

Copy link
Contributor

@whyitfor whyitfor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reflected on this one a bit, and discussed with @rbs-jacob.

Let's make the following changes.

  1. Remove the check-python-reqs argument from build-image.py
  2. build-image.py should add an inspect target to the Makefile it generates that runs python3 -m pip check. This will enable users who want to check for this consistency to do so.
  3. Does --network="none" work for all use cases? If so, let's make it default. If not, we should remove it.

@ANogin ANogin marked this pull request as draft January 15, 2025 20:50
@ANogin ANogin changed the title Add --check_python_reqs option to build_image.py Ensure no missing pip dependencies in build_image.py Jan 15, 2025
@ANogin
Copy link
Collaborator Author

ANogin commented Jan 15, 2025

I've reflected on this one a bit, and discussed with @rbs-jacob.

Let's make the following changes.

  1. Remove the check-python-reqs argument from build-image.py
  2. build-image.py should add an inspect target to the Makefile it generates that runs python3 -m pip check. This will enable users who want to check for this consistency to do so.
  3. Does --network="none" work for all use cases? If so, let's make it default. If not, we should remove it.

Done, should be good to go once #419 is fixed.

build_image.py Outdated Show resolved Hide resolved
Comment on lines +10 to +12
# Legacy-edinable is needed to allow mypy to find packages.
# See https://github.com/python/mypy/issues/13392
SETUPTOOLS_ENABLE_FEATURES="legacy-editable" $(PIP) install -e .[test]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why this change is needed.

#561 updated these targets to enable editable_mode=compat -- is this not the same issue?

cc @alchzh

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes this change is no longer needed (the command line flag is preferred over the environment variable)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, with command line flag it insists on reaching out to pypi over network, but with the environment variable it does not.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've looked into this a bit and reflected.

If I understand correctly,SETUPTOOLS_ENABLE_FEATURES results in using a pre-PEP 517 backend. editable_mode=compat uses a PEP660 backend that provides some backwards compatibility. Switching back to SETUPTOOLS_ENABLE_FEATURES does not make sense to me: is moving the repo in the wrong direction, away from PEP660.

It looks like there are ways to get pip to not reach out to the network (a combination of --no-index and --find-links) which we might want to eventually look into.

For now though, I'd like to prioritize getting the pip check functionality this MR adds in, without the --network="none" changes.

@ANogin ANogin requested a review from whyitfor January 16, 2025 01:16
@ANogin ANogin marked this pull request as ready for review January 21, 2025 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants