LOCAL_VIRTUALENV.rst

• Local virtualenv environment
  • Installation
  • Installing other extras
  • Running individual tests
  • Running tests directly from the IDE
  • Running integration tests

Local virtualenv environment

Installation

Install Python (3.5 or 3.6), MySQL, and libxml by using system-level package managers like yum, apt-get for Linux, or Homebrew for Mac OS at first. Refer to the Dockerfile for a comprehensive list of required packages.

In order to use your IDE you need you can use the virtual environment. Ideally you should setup virtualenv for all python versions that Airflow supports (3.5, 3.6). An easy way to create the virtualenv is to use virtualenvwrapper - it allows you to easily switch between virtualenvs using workon command and mange your virtual environments more easily. Typically creating the environment can be done by:

mkvirtualenv <ENV_NAME> --python=python<VERSION>

Then you need to install python PIP requirements. Typically it can be done with: pip install -e ".[devel]".

After creating the virtualenv, run this command to create the Airflow sqlite database:

airflow db init

Creating virtualenv can be automated with Breeze environment

Once initialization is done, you should select the virtualenv you initialized as the project's default virtualenv in your IDE.

After setting it up - you can use the usual "Run Test" option of the IDE and have the autocomplete and documentation support from IDE as well as you can debug and view the sources of Airflow - which is very helpful during development.

Installing other extras

You can also other extras (like [mysql], [gcp] etc. via pip install -e [EXTRA1,EXTRA2 ...]. However some of the extras have additional system requirements and you might need to install additional packages on your local machine.

For example if you have trouble installing mysql client on MacOS and you have an error similar to

ld: library not found for -lssl

you should set LIBRARY_PATH before running pip install:

export LIBRARY_PATH=$LIBRARY_PATH:/usr/local/opt/openssl/lib/

The full list of extras is available in setup.py

Running individual tests

Once you activate virtualenv (or enter docker container) as described below you should be able to run run-tests at will (it is in the path in Docker environment but you need to prepend it with ./ in local virtualenv (./run-tests).

Note that this script has several flags that can be useful for your testing.

Usage: run-tests [FLAGS] [TESTS_TO_RUN] -- <EXTRA_NOSETEST_ARGS>

Runs tests specified (or all tests if no tests are specified)

Flags:

-h, --help
        Shows this help message.

-i, --with-db-init
        Forces database initialization before tests

-s, --nocapture
        Don't capture stdout when running the tests. This is useful if you are
        debugging with ipdb and want to drop into console with it
        by adding this line to source code:

            import ipdb; ipdb.set_trace()

-v, --verbose
        Verbose output showing coloured output of tests being run and summary
        of the tests - in a manner similar to the tests run in the CI environment.

You can pass extra parameters to nose, by adding nose arguments after --

For example, in order to just execute the "core" unit tests and add ipdb set_trace method, you can run the following command:

./run-tests tests.core:TestCore --nocapture --verbose

or a single test method without colors or debug logs:

./run-tests tests.core:TestCore.test_check_operators

Note that ./run_tests script runs tests but the first time it runs, it performs database initialisation. If you run further tests without leaving the environment, the database will not be initialized, but you can always force database initialization with --with-db-init (-i) switch. The scripts will inform you what you can do when they are run.

Running tests directly from the IDE

Once you configure your tests to use the virtualenv you created. running tests from IDE is as simple as:

Running integration tests

Note that while most of the tests are typical "unit" tests that do not require external components, there are a number of tests that are more of "integration" or even "system" tests. You can technically use local virtualenv to run those tests, but it requires to setup a number of external components (databases/queues/kubernetes and the like) so it is much easier to use the Breeze development environment for those tests.

Note - soon we will separate the integration and system tests out so that you can clearly know which tests are unit tests and can be run in the local virtualenv and which should be run using Breeze.