Install pip dependency from a repo

[Joshua3212]

Hello everyone,

Disclaimer: I am pretty new to airflow (so I might be missingGitHubhing)

Goal: Install a Github repository as a pip package for use in DAGs.
My Airflow instance is running using docker-compose.

What I tried: Airflow seems to offer a few different ways to install pip packages

1. add the repository via the _PIP_ADDITIONAL_REQUIREMENTS envvar like so:

_PIP_ADDITIONAL_REQUIREMENTS='git+https://github.com/hudduapp/dummy-sdk.git#egg=dummy'

Result: the container crashes with no error shown

2. build the docker image again via a custom Dockerfile:

FROM apache/airflow:latest
USER root
COPY ./requirements.txt requirements.txt
RUN pip install --user --upgrade pip
RUN pip install --user -r requirements.txt

Result:

You are running pip as root. Please use 'airflow' user to run pip!
See: https://airflow.apache.org/docs/docker-stack/build.html#adding-a-new-pypi-package

But if I try airflow as the USER instead of root I get this beauty:

Collecting dummy
Cloning https://github.com/hudduapp/dummy-sdk.git to /tmp/pip-install-zgf0juzs/dummy_e3cdf232eaae4431ad66388c68de7420
ERROR: Error [Errno 13] Permission denied: 'git' while executing command git version
ERROR: No permission to execute 'git' - install it locally, globally (ask admin), or check your PATH. See possible solutions at > https://pip.pypa.io/en/latest/reference/pip_freeze/#fixing-permission-denied.
The command '/bin/bash -o pipefail -o errexit -o nounset -o nolog -c pip install --user -r requirements.txt' returned a non-zero code: 1
ERROR: Service 'airflow-init' failed to build : Build failed

I hope it's possible to achieve this since using GitHub to host packages can be pretty convenient.

[potiuk]

If you want to use "git" - you need to install it first. Airlfow PROD image simply does not have git installed. you need to install it first following the other examples (about installing system packages)

The error you get is very clear about it (and this is an error of pip not airflow, so if it is not clear enough, you should raise it in `pip' not in airflow repo.

ERROR: No permission to execute 'git' - install it locally, globally (ask admin), or check your PATH. 
See possible solutions at > https://pip.pypa.io/en/latest/reference/pip_freeze/#fixing-permission-denied.

[Joshua3212]

You're right!
My apologies.