Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

package-requirements always returns exit code 0, even on failure #388

Open
matthewbarreiro opened this issue Aug 19, 2024 · 1 comment · May be fixed by #389
Open

package-requirements always returns exit code 0, even on failure #388

matthewbarreiro opened this issue Aug 19, 2024 · 1 comment · May be fixed by #389

Comments

@matthewbarreiro
Copy link

Running ./mwaa-local-env package-requirements will always result in an exit code of 0, even on failure.

Steps to Reproduce:
Note: All steps were performed using v2.8.1, but the behavior should be the same for all versions that include this function (v2.2.2+).

  1. Modify requirements.txt to force a failure. E.g. add apache-airflow-providers-datadog==3.5.0 to the requirements.txt included in the repo.

  2. Run ./mwaa-local-env package-requirements and check exit code.

    Click here for shell output
     ➜  aws-mwaa-local-runner git:(v2.8.1) ✗ ./mwaa-local-env package-requirements 
    Container amazon/mwaa-local:2_8 exists. Skipping build
    Packaging requirements.txt into plugins
    Collecting apache-airflow-providers-snowflake==5.2.1
      Downloading apache_airflow_providers_snowflake-5.2.1-py3-none-any.whl (37 kB)
    Collecting apache-airflow-providers-mysql==5.5.1
      Downloading apache_airflow_providers_mysql-5.5.1-py3-none-any.whl (25 kB)
    ERROR: Cannot install apache-airflow-providers-datadog==3.5.0 because these package versions have conflicting dependencies.
    
    The conflict is caused by:
        The user requested apache-airflow-providers-datadog==3.5.0
        The user requested (constraint) apache-airflow-providers-datadog==3.5.1
    
    To fix this you could try to:
    1. loosen the range of package versions you've specified
    4. remove package versions to allow pip attempt to solve the dependency conflict
    
    ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    
    [notice] A new release of pip available: 22.3.1 -> 24.2
    [notice] To update, run: pip install --upgrade pip
    updating: README.md (deflated 37%)
    
     ➜  aws-mwaa-local-runner git:(v2.8.1) ✗ echo $?
    0

Note that ./mwaa-local-env test-requirements does have a proper non-zero exit code on failure. However both commands should have a non-zero exit code on failure.

Click here for shell output
 ➜  aws-mwaa-local-runner git:(v2.8.1) ✗ ./mwaa-local-env test-requirements    
Container amazon/mwaa-local:2_8 exists. Skipping build
--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.8.1/constraints-3.11.txt"
Installing requirements.txt
Collecting apache-airflow-providers-snowflake==5.2.1
  Downloading apache_airflow_providers_snowflake-5.2.1-py3-none-any.whl (37 kB)
Collecting apache-airflow-providers-mysql==5.5.1
  Downloading apache_airflow_providers_mysql-5.5.1-py3-none-any.whl (25 kB)
ERROR: Cannot install apache-airflow-providers-datadog==3.5.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested apache-airflow-providers-datadog==3.5.0
    The user requested (constraint) apache-airflow-providers-datadog==3.5.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[notice] A new release of pip available: 22.3.1 -> 24.2
[notice] To update, run: pip install --upgrade pip

 ➜  aws-mwaa-local-runner git:(v2.8.1) ✗ echo $?                            
1

Cause:
The issue stems from the package_requirements() function in entrypoint.sh. Specifically, there is no check that pip3 download -r ... completed successfully. The script will just attempt to zip whatever was downloaded and continue on it's way.

Fix:
I'm working on a fix for this and will open a PR to v2.9.2 when it is ready. I kindly request the fix be merged to lower branches (either all v2.2.2+ or at least to the branches for versions that are still supported).

Thank you!

@matthewbarreiro
Copy link
Author

On a related note - my fix will only be for pip3 download failures. It may be worth considering using set -e or similar for this and similar commands, but that is presumably a larger, internal conversation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant